Oct 29 23:12:37.816746 ip-10-0-133-100 systemd[1]: Starting Kubernetes Kubelet... Oct 29 23:12:38.218895 ip-10-0-133-100 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 23:12:38.218895 ip-10-0-133-100 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 23:12:38.218895 ip-10-0-133-100 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 23:12:38.218895 ip-10-0-133-100 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 23:12:38.223260 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223171 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 23:12:38.223260 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223259 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 23:12:38.223260 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223263 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 23:12:38.223260 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223266 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223269 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223272 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223276 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223278 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223281 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223284 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223286 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223289 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223291 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223294 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223297 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223300 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223303 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223306 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223308 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223317 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223320 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223322 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 23:12:38.223402 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223325 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223327 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223330 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223333 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223335 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223338 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223340 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223343 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223346 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223348 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223351 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223355 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223359 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223363 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223366 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223369 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223372 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223375 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223377 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223380 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 23:12:38.223876 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223383 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223386 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223389 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223392 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223395 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223397 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223400 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223403 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223406 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223410 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223412 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223415 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223418 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223421 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223424 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223426 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223429 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223431 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223434 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223436 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 23:12:38.224344 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223439 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223441 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223444 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223446 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223449 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223451 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223466 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223471 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223474 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223477 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223480 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223482 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223485 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223488 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223490 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223493 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223496 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223498 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223501 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223504 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 23:12:38.224868 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223507 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223509 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223512 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223514 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223517 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223520 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223522 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223524 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223527 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223532 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223536 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223539 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223542 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223545 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223548 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223551 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223554 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223556 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223559 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 23:12:38.225343 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223562 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223565 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223567 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223570 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.223573 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225269 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225278 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225282 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225285 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225289 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225292 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225295 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225299 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225303 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225307 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225309 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225312 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225314 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225317 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225319 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 23:12:38.225803 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225322 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225324 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225327 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225329 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225332 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225335 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225337 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225340 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225343 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225346 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225348 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225351 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225354 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225357 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225360 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225362 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225365 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225368 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225370 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 23:12:38.226279 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225373 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225375 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225378 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225381 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225384 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225386 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225388 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225391 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225393 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225396 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225398 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225401 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225403 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225406 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225408 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225411 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225413 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225415 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225418 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225420 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 23:12:38.226751 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225423 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225425 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225428 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225431 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225433 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225437 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225439 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225442 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225444 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225447 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225450 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225453 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225470 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225472 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225475 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225478 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225481 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225484 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225486 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 23:12:38.227213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225489 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225491 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225494 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225496 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225499 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225501 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225504 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225506 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225509 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225512 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225514 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225518 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225520 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225523 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225526 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225530 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225534 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225537 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225540 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 23:12:38.227680 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225543 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225546 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225548 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225551 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225553 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225557 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225559 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225562 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225564 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225567 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225569 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225572 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225575 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.225578 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226081 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226085 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226088 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226091 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226093 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226096 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 23:12:38.228124 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226099 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226102 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226104 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226106 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226109 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226112 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226115 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226117 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226120 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226123 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226126 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226128 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226131 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226134 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226136 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226139 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226141 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226144 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226147 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 23:12:38.228625 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226149 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226152 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226162 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226164 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226168 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226171 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226173 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226176 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226178 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226181 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226183 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226186 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226188 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226191 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226194 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226196 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226199 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226202 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226204 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 23:12:38.229072 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226207 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226209 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226212 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226214 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226217 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226220 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226223 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226225 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226228 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226230 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226234 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226238 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226240 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226243 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226246 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226248 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226251 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226253 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226256 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 23:12:38.229550 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226259 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226262 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226264 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226268 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226271 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226275 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226277 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226280 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226283 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226286 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226288 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226291 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226294 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226296 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226299 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226301 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226304 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226306 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226310 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226312 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 23:12:38.230047 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226315 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226317 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226320 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226322 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226326 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226328 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226331 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226333 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226336 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226338 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226341 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226343 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226346 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226349 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226351 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226354 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226356 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226359 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226361 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226363 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 23:12:38.230536 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226366 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226371 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.226373 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226490 2571 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226498 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226505 2571 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226510 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226514 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226517 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226522 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226526 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226530 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226533 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226537 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226540 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226543 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226547 2571 flags.go:64] FLAG: --cgroup-root="" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226550 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226553 2571 flags.go:64] FLAG: --client-ca-file="" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226556 2571 flags.go:64] FLAG: --cloud-provider="external" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226559 2571 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226564 2571 flags.go:64] FLAG: --cluster-domain="" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226567 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 23:12:38.231115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226570 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226573 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226577 2571 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226581 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226584 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226587 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226591 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226594 2571 flags.go:64] FLAG: --contention-profiling="false" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226597 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226600 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226606 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226609 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226614 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226617 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226620 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226623 2571 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226626 2571 flags.go:64] FLAG: --enable-server="true" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226630 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226634 2571 flags.go:64] FLAG: --event-burst="100" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226637 2571 flags.go:64] FLAG: --event-qps="50" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226640 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226643 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226646 2571 flags.go:64] FLAG: --eviction-hard="" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226651 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 23:12:38.231722 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226654 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226657 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226660 2571 flags.go:64] FLAG: --eviction-soft="" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226663 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226666 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226670 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226673 2571 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226676 2571 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226679 2571 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226682 2571 flags.go:64] FLAG: --feature-gates="" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226689 2571 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226693 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226696 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226699 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226703 2571 flags.go:64] FLAG: --healthz-port="10248" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226706 2571 flags.go:64] FLAG: --help="false" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226709 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226712 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226718 2571 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226720 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226724 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226727 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226730 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 23:12:38.232308 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226733 2571 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226736 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226739 2571 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226742 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226745 2571 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226748 2571 flags.go:64] FLAG: --kube-reserved="" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226751 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226754 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226757 2571 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226760 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226763 2571 flags.go:64] FLAG: --lock-file="" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226766 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226769 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226772 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226778 2571 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226781 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226784 2571 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226787 2571 flags.go:64] FLAG: --logging-format="text" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226790 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226793 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226796 2571 flags.go:64] FLAG: --manifest-url="" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226800 2571 flags.go:64] FLAG: --manifest-url-header="" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226804 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226807 2571 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226811 2571 flags.go:64] FLAG: --max-pods="110" Oct 29 23:12:38.232877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226814 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226818 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226823 2571 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226826 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226829 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226832 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226835 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226842 2571 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226845 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226849 2571 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226852 2571 flags.go:64] FLAG: --pod-cidr="" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226855 2571 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226858 2571 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226861 2571 flags.go:64] FLAG: --pods-per-core="0" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226864 2571 flags.go:64] FLAG: --port="10250" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226867 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226870 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03d251752a696b359" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226873 2571 flags.go:64] FLAG: --qos-reserved="" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226876 2571 flags.go:64] FLAG: --read-only-port="10255" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226879 2571 flags.go:64] FLAG: --register-node="true" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226882 2571 flags.go:64] FLAG: --register-with-taints="" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226886 2571 flags.go:64] FLAG: --registry-burst="10" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226889 2571 flags.go:64] FLAG: --registry-qps="5" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226892 2571 flags.go:64] FLAG: --reserved-cpus="" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226895 2571 flags.go:64] FLAG: --reserved-memory="" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226899 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 23:12:38.233579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226902 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226905 2571 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226908 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226911 2571 flags.go:64] FLAG: --runonce="false" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226914 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226917 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226920 2571 flags.go:64] FLAG: --seccomp-default="false" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226923 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226929 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226932 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226935 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226939 2571 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226942 2571 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226944 2571 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226947 2571 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226950 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226953 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226956 2571 flags.go:64] FLAG: --system-cgroups="" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226959 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226964 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226967 2571 flags.go:64] FLAG: --tls-cert-file="" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226970 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226975 2571 flags.go:64] FLAG: --tls-min-version="" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226977 2571 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226980 2571 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 23:12:38.234671 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226983 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226986 2571 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226989 2571 flags.go:64] FLAG: --v="2" Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226993 2571 flags.go:64] FLAG: --version="false" Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.226997 2571 flags.go:64] FLAG: --vmodule="" Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.227001 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.227005 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227142 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227146 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227149 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227152 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227155 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227158 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227160 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227163 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227168 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227171 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227173 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227176 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227179 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227181 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 23:12:38.235627 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227184 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227186 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227189 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227191 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227194 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227197 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227199 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227202 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227205 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227207 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227210 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227212 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227215 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227217 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227222 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227225 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227229 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227232 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227235 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 23:12:38.236192 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227238 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227240 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227243 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227245 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227248 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227251 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227254 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227258 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227261 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227263 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227266 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227269 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227271 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227274 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227276 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227279 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227281 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227284 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227287 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227289 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 23:12:38.236668 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227291 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227294 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227297 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227299 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227302 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227305 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227308 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227310 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227313 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227315 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227318 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227321 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227323 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227326 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227328 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227331 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227333 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227338 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227342 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227350 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 23:12:38.237136 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227353 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227356 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227358 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227361 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227364 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227366 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227369 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227372 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227374 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227377 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227379 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227382 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227385 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227387 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227390 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227392 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227395 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227397 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227400 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 23:12:38.237632 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227403 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 23:12:38.238085 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227405 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 23:12:38.238085 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227408 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 23:12:38.238085 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227411 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 23:12:38.238085 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227413 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 23:12:38.238085 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227416 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 23:12:38.238085 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227418 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 23:12:38.238085 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227421 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 23:12:38.238085 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227423 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 23:12:38.238085 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227426 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 23:12:38.238085 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227428 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 23:12:38.238085 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227431 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 23:12:38.238085 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227436 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 23:12:38.238085 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227439 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 23:12:38.238085 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.227478 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 23:12:38.238085 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227519 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 23:12:38.238085 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227524 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227526 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227529 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227532 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227535 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227538 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227540 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227543 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227546 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227548 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227551 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227553 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227556 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227559 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227562 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227564 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227567 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227570 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227572 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 23:12:38.238490 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227575 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227577 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227580 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227582 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227585 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227588 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227590 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227593 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227595 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227600 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227605 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227607 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227610 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227612 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227616 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227620 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227624 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227627 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227629 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 23:12:38.238937 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227632 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227635 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227637 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227640 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227642 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227646 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227648 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227651 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227654 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227656 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227659 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227661 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227664 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227666 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227669 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227672 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227674 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227677 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227679 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227682 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 23:12:38.239414 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227685 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227687 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227691 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227695 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227698 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227701 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227703 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227706 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227708 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227712 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227714 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227717 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227719 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227722 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227724 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227727 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227729 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227732 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227735 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227738 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 23:12:38.240040 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227741 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227744 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227747 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227749 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227752 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227754 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227757 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227760 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227763 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227765 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227767 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227770 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227772 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227775 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227778 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227780 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227783 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227786 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227788 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 23:12:38.240523 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227791 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227794 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227796 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227799 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227801 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227804 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227806 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.227809 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.236766 2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.236787 2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.237599 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.237608 2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237669 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237676 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237682 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237687 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237691 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237693 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237696 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237699 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 23:12:38.240967 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237702 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237705 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237708 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237710 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237713 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237716 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237719 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237721 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237724 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237727 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237729 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237732 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237735 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237737 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237742 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237745 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237748 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237751 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237756 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 23:12:38.241504 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237761 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237765 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237767 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237770 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237773 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237775 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237778 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237780 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237783 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237785 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237788 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237791 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237793 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237796 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237799 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237802 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237804 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237807 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237809 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237812 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 23:12:38.241969 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237816 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237821 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237824 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237828 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237832 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237837 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237841 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237845 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237847 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237850 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237852 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237855 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237857 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237860 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237862 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237865 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237867 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237870 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237873 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 23:12:38.242489 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237875 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237878 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237880 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237883 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237885 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237888 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237891 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237894 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237896 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237899 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237902 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237905 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237909 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237923 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237928 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237932 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237934 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237937 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237940 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 23:12:38.242986 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237942 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237945 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237947 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237950 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237952 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237955 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237957 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237960 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237962 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237965 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237967 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237970 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237972 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237975 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237977 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237981 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237985 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237989 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237993 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237996 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 23:12:38.243604 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.237999 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 23:12:38.244100 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.238029 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 23:12:38.244100 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238197 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 23:12:38.244100 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238203 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 23:12:38.244100 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238206 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 23:12:38.244100 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238209 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 23:12:38.244100 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238213 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 23:12:38.244100 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238219 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 23:12:38.244100 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238223 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 23:12:38.244100 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238227 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 23:12:38.244100 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238230 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 23:12:38.244100 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238233 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 23:12:38.244100 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238235 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 23:12:38.244100 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238238 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 23:12:38.244100 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238241 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 23:12:38.244100 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238243 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 23:12:38.244100 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238246 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238248 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238251 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238253 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238256 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238258 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238261 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238264 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238266 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238269 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238271 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238274 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238277 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238279 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238282 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238285 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238287 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238290 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238294 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238299 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 23:12:38.244499 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238303 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238306 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238308 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238311 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238315 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238317 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238320 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238323 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238326 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238328 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238331 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238334 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238336 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238339 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238341 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238344 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238346 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238349 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238351 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238354 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 23:12:38.245002 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238356 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238359 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238361 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238364 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238367 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238371 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238376 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238380 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238383 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238385 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238388 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238390 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238396 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238399 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238402 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238404 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238409 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238413 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238416 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 23:12:38.245610 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238418 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238421 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238423 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238426 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238429 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238431 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238434 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238437 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238441 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238444 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238449 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238453 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238468 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238471 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238474 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238476 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238479 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238482 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238485 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 23:12:38.246379 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238488 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 23:12:38.246947 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238491 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 23:12:38.246947 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238493 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 23:12:38.246947 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238496 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 23:12:38.246947 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238499 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 23:12:38.246947 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238501 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 23:12:38.246947 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238504 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 23:12:38.246947 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238507 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 23:12:38.246947 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238509 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 23:12:38.246947 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238513 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 23:12:38.246947 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238515 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 23:12:38.246947 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238518 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 23:12:38.246947 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238522 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 23:12:38.246947 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238526 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 23:12:38.246947 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.238567 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 23:12:38.246947 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238608 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 23:12:38.246947 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238614 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238618 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238622 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238625 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238628 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238632 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238635 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238637 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238640 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238643 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238646 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238648 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238651 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238653 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238656 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238658 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238661 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238664 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238666 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 23:12:38.247400 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238669 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238671 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238674 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238677 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238681 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238686 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238689 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238692 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238695 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238698 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238701 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238704 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238707 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238709 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238712 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238715 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238717 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238720 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238722 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 23:12:38.247890 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238725 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238727 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238730 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238732 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238734 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238737 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238739 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238742 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238744 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238747 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238749 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238752 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238755 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238760 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238764 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238768 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238770 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238773 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238775 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 23:12:38.248351 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238778 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238781 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238783 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238786 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238788 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238792 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238794 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238797 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238799 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238802 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238804 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238807 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238810 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238812 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238815 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238818 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238820 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238823 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238825 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238827 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 23:12:38.249091 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238831 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238835 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238840 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238844 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238847 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238849 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238852 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238854 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238857 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238859 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238862 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238864 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238867 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238869 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238872 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238874 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238877 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238880 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238883 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 23:12:38.249658 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238886 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 23:12:38.250120 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238891 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 23:12:38.250120 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238894 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 23:12:38.250120 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238898 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 23:12:38.250120 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238901 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 23:12:38.250120 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238904 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 23:12:38.250120 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238909 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 23:12:38.250120 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238913 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 23:12:38.250120 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:38.238917 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 23:12:38.250120 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.238982 2571 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 23:12:38.250120 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.241781 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 29 23:12:38.250120 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.242800 2571 server.go:1014] "Starting client certificate rotation" Oct 29 23:12:38.250120 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.242907 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 23:12:38.250120 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.242968 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 23:12:38.250120 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.244885 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 23:12:38.250120 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.246030 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 23:12:38.257034 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.257018 2571 log.go:25] "Validated CRI v1 runtime API" Oct 29 23:12:38.260467 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.260437 2571 log.go:25] "Validated CRI v1 image API" Oct 29 23:12:38.261351 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.261328 2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 23:12:38.268184 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.268159 2571 fs.go:135] Filesystem UUIDs: map[704d1010-6224-4aee-9bb5-357fc0ba23f7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7b96e30c-11d1-4091-ae3c-9d8dc4d51008:/dev/nvme0n1p3] Oct 29 23:12:38.268267 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.268185 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 29 23:12:38.274387 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.274243 2571 manager.go:217] Machine: {Timestamp:2026-10-29 23:12:38.272844548 +0000 UTC m=+0.344373075 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3044130 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22f7b8284ee5bb1ac59f8855e8709b SystemUUID:ec22f7b8-284e-e5bb-1ac5-9f8855e8709b BootID:cb4ecad0-0bdb-4e9f-a9e1-abcfb0f4534e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:8a:47:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:8a:47:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:f8:77:9d:f8:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 23:12:38.274387 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.274378 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 23:12:38.274586 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.274517 2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 23:12:38.277158 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.277130 2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 23:12:38.277778 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.277168 2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-100.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 23:12:38.277912 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.277781 2571 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 23:12:38.277912 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.277809 2571 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 23:12:38.277912 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.277829 2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 23:12:38.278585 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.278565 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 23:12:38.279492 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.279474 2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 23:12:38.283210 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.283198 2571 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 23:12:38.283335 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.283326 2571 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 23:12:38.284046 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.284034 2571 kubelet.go:493] "Attempting to sync node with API server" Oct 29 23:12:38.284081 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.284057 2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 23:12:38.284081 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.284068 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 23:12:38.284081 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.284078 2571 kubelet.go:405] "Adding apiserver pod source" Oct 29 23:12:38.284190 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.284087 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 23:12:38.284494 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.284479 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 23:12:38.286801 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.286786 2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 23:12:38.291824 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.291808 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 23:12:38.292710 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.292694 2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 23:12:38.292801 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.292736 2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 23:12:38.292946 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.292933 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 23:12:38.293004 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.292953 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 23:12:38.293004 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.292962 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 23:12:38.293004 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.292972 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 23:12:38.293004 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.292982 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 23:12:38.293004 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.292992 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 23:12:38.293550 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.293540 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 23:12:38.293599 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.293555 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 23:12:38.293599 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.293566 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 23:12:38.293599 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.293577 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 23:12:38.293599 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.293591 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 23:12:38.293744 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.293605 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 23:12:38.294407 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.294396 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 23:12:38.294477 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.294410 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 23:12:38.298256 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.298240 2571 server.go:1263] "Started kubelet" Oct 29 23:12:38.298340 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.298299 2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 23:12:38.298414 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.298344 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 23:12:38.298475 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.298431 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 23:12:38.299060 ip-10-0-133-100 systemd[1]: Started Kubernetes Kubelet. Oct 29 23:12:38.299914 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.299896 2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 23:12:38.300943 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.300926 2571 server.go:317] "Adding debug handlers to kubelet server" Oct 29 23:12:38.303735 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.303710 2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 29 23:12:38.305394 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.305372 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 23:12:38.305883 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.305864 2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 23:12:38.306028 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.306007 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:12:38.306431 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.306413 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 23:12:38.306431 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.306414 2571 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 23:12:38.306591 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.306445 2571 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 23:12:38.306591 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.306528 2571 reconstruct.go:97] "Volume reconstruction finished" Oct 29 23:12:38.306591 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.306538 2571 reconciler.go:29] "Reconciler: start to sync state" Oct 29 23:12:38.306839 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.306825 2571 factory.go:55] Registering systemd factory Oct 29 23:12:38.306875 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.306843 2571 factory.go:223] Registration of the systemd container factory successfully Oct 29 23:12:38.307067 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.307053 2571 factory.go:153] Registering CRI-O factory Oct 29 23:12:38.307135 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.307069 2571 factory.go:223] Registration of the crio container factory successfully Oct 29 23:12:38.307135 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.307117 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 23:12:38.307238 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.307145 2571 factory.go:103] Registering Raw factory Oct 29 23:12:38.307238 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.307160 2571 manager.go:1199] Started watching for new ooms in manager Oct 29 23:12:38.308017 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.308002 2571 manager.go:319] Starting recovery of all containers Oct 29 23:12:38.321347 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.321327 2571 manager.go:324] Recovery completed Oct 29 23:12:38.325589 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.325574 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 23:12:38.327569 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.327547 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 23:12:38.327647 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.327581 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 23:12:38.327647 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.327592 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 23:12:38.328159 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.328144 2571 cpu_manager.go:226] "Starting" policy="none" Oct 29 23:12:38.328159 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.328158 2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 23:12:38.328266 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.328177 2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 23:12:38.328266 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.328144 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 23:12:38.328996 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.328983 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 23:12:38.329091 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.329008 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 23:12:38.329091 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.329017 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 23:12:38.330500 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.330489 2571 policy_none.go:50] "Start" Oct 29 23:12:38.330534 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.330504 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 23:12:38.330534 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.330523 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 23:12:38.332502 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.332475 2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 29 23:12:38.332598 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.332512 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-100.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 29 23:12:38.332667 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.332644 2571 policy_none.go:44] "Start" Oct 29 23:12:38.332813 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.332790 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 23:12:38.333575 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.332590 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-100.ec2.internal.18e322ffb68be299 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-100.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-100.ec2.internal,},FirstTimestamp:2026-10-29 23:12:38.298215065 +0000 UTC m=+0.369743592,LastTimestamp:2026-10-29 23:12:38.298215065 +0000 UTC m=+0.369743592,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-100.ec2.internal,}" Oct 29 23:12:38.335668 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.335650 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cwr4s" Oct 29 23:12:38.342942 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.342924 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cwr4s" Oct 29 23:12:38.347001 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.346923 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-100.ec2.internal.18e322ffb84bc6ce default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-100.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-100.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-100.ec2.internal,},FirstTimestamp:2026-10-29 23:12:38.327568078 +0000 UTC m=+0.399096605,LastTimestamp:2026-10-29 23:12:38.327568078 +0000 UTC m=+0.399096605,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-100.ec2.internal,}" Oct 29 23:12:38.347107 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.346985 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 29 23:12:38.356921 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.356893 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 29 23:12:38.356921 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.356912 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 29 23:12:38.388561 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.369410 2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 23:12:38.388561 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.372247 2571 manager.go:341] "Starting Device Plugin manager" Oct 29 23:12:38.388561 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.372293 2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 23:12:38.388561 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.372306 2571 server.go:87] "Starting device plugin registration server" Oct 29 23:12:38.388561 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.372609 2571 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 23:12:38.388561 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.372627 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 23:12:38.388561 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.372817 2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 23:12:38.388561 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.372900 2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 23:12:38.388561 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.372908 2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 23:12:38.388561 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.373704 2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 23:12:38.388561 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.373752 2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:12:38.388561 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.376981 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 23:12:38.388561 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.378284 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 23:12:38.388561 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.378304 2571 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 23:12:38.388561 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.378322 2571 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 23:12:38.388561 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.378355 2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 29 23:12:38.393800 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.393777 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 23:12:38.473757 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.473683 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 23:12:38.474723 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.474705 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 23:12:38.474811 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.474736 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 23:12:38.474811 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.474746 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 23:12:38.474811 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.474769 2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.478898 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.478864 2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal"] Oct 29 23:12:38.479008 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.478937 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 23:12:38.479941 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.479777 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 23:12:38.480095 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.479953 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 23:12:38.480095 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.479965 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 23:12:38.482580 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.482565 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 23:12:38.482735 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.482719 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.482788 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.482750 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 23:12:38.483328 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.483304 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 23:12:38.483328 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.483331 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 23:12:38.483497 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.483313 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 23:12:38.483497 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.483367 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 23:12:38.483497 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.483344 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 23:12:38.483497 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.483383 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 23:12:38.485627 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.485611 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.485715 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.485645 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 23:12:38.486403 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.486387 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 23:12:38.486494 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.486413 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 23:12:38.486494 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.486424 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 23:12:38.494951 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.494936 2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.495026 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.494956 2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-100.ec2.internal\": node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:12:38.507409 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.507373 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/563071327f3cc335e24937880997d7f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal\" (UID: \"563071327f3cc335e24937880997d7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.507409 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.507404 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a7e411141454709a32dcea46ca88fb7-config\") pod \"kube-apiserver-proxy-ip-10-0-133-100.ec2.internal\" (UID: \"3a7e411141454709a32dcea46ca88fb7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.507572 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.507422 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/563071327f3cc335e24937880997d7f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal\" (UID: \"563071327f3cc335e24937880997d7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.512041 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.512017 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-100.ec2.internal\" not found" node="ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.514714 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.514693 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-100.ec2.internal\" not found" node="ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.546323 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.546296 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:12:38.608474 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.608431 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/563071327f3cc335e24937880997d7f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal\" (UID: \"563071327f3cc335e24937880997d7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.608613 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.608478 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/563071327f3cc335e24937880997d7f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal\" (UID: \"563071327f3cc335e24937880997d7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.608613 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.608516 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/563071327f3cc335e24937880997d7f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal\" (UID: \"563071327f3cc335e24937880997d7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.608613 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.608518 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a7e411141454709a32dcea46ca88fb7-config\") pod \"kube-apiserver-proxy-ip-10-0-133-100.ec2.internal\" (UID: \"3a7e411141454709a32dcea46ca88fb7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.608613 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.608553 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a7e411141454709a32dcea46ca88fb7-config\") pod \"kube-apiserver-proxy-ip-10-0-133-100.ec2.internal\" (UID: \"3a7e411141454709a32dcea46ca88fb7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.608613 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.608587 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/563071327f3cc335e24937880997d7f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal\" (UID: \"563071327f3cc335e24937880997d7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.647161 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.647119 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:12:38.748095 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.748017 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:12:38.816328 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.816300 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.818468 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:38.818442 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" Oct 29 23:12:38.848851 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.848820 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:12:38.949338 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:38.949284 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:12:39.050731 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:39.050645 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:12:39.153470 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:39.153426 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:12:39.242443 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.242408 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 29 23:12:39.243110 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.242543 2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 23:12:39.243110 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.242566 2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 29 23:12:39.243110 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.242574 2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 29 23:12:39.254554 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:39.254524 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:12:39.348014 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.347963 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 23:07:38 +0000 UTC" deadline="2028-07-06 03:59:15.076172185 +0000 UTC" Oct 29 23:12:39.348014 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.348011 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14764h46m35.728166946s" Oct 29 23:12:39.354710 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:39.354681 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:12:39.437882 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:39.437841 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7e411141454709a32dcea46ca88fb7.slice/crio-fbe5b256b4f104384b21ffe6e16f9f413a15378154a8980529b4044551b908a4 WatchSource:0}: Error finding container fbe5b256b4f104384b21ffe6e16f9f413a15378154a8980529b4044551b908a4: Status 404 returned error can't find the container with id fbe5b256b4f104384b21ffe6e16f9f413a15378154a8980529b4044551b908a4 Oct 29 23:12:39.447662 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.447639 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:12:39.453672 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:39.453648 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563071327f3cc335e24937880997d7f4.slice/crio-eb5fcf7da1f60dc265c4cec8b0771d99a3915acbb6d6488ee5bb7fb65298f379 WatchSource:0}: Error finding container eb5fcf7da1f60dc265c4cec8b0771d99a3915acbb6d6488ee5bb7fb65298f379: Status 404 returned error can't find the container with id eb5fcf7da1f60dc265c4cec8b0771d99a3915acbb6d6488ee5bb7fb65298f379 Oct 29 23:12:39.454779 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:39.454757 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:12:39.487440 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.487406 2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 23:12:39.525185 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.525151 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 23:12:39.530117 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.530098 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 23:12:39.530209 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.530134 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 23:12:39.530209 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.530144 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 23:12:39.555791 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:39.555763 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:12:39.658762 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:39.658697 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:12:39.759277 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:39.759246 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:12:39.853916 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.853875 2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 23:12:39.906474 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.906428 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:12:39.936213 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.936027 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 23:12:39.937324 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.937301 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" Oct 29 23:12:39.960921 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:39.960877 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 23:12:40.285373 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.285278 2571 apiserver.go:52] "Watching apiserver" Oct 29 23:12:40.307390 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.307352 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 29 23:12:40.333633 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.333600 2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 23:12:40.334029 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.334005 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-f456x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal","openshift-multus/multus-additional-cni-plugins-fn78d","kube-system/global-pull-secret-syncer-xcszc","openshift-network-operator/iptables-alerter-twqrq","kube-system/konnectivity-agent-klctw","kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal","openshift-cluster-node-tuning-operator/tuned-ccm62","openshift-image-registry/node-ca-tr7bl","openshift-multus/multus-ltlxg","openshift-multus/network-metrics-daemon-xwn99","openshift-network-diagnostics/network-check-target-69jbt","openshift-ovn-kubernetes/ovnkube-node-xgnjg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt"] Oct 29 23:12:40.336980 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.336950 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:40.337086 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:40.337044 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:12:40.339430 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.339404 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.343203 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.343177 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.345635 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.345616 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f456x" Oct 29 23:12:40.348131 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.348101 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:12:40.350609 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.350587 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-klctw" Oct 29 23:12:40.353616 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.353157 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.356489 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.356453 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:40.356614 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:40.356540 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:12:40.358848 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.358825 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 23:12:40.358981 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.358922 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 23:12:40.358981 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.358922 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 23:12:40.359108 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.359044 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qbkzf\"" Oct 29 23:12:40.360024 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.360004 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hwcbb\"" Oct 29 23:12:40.360162 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.360134 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 23:12:40.360235 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.360212 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 23:12:40.360357 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.360339 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 23:12:40.360536 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.360515 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 23:12:40.360627 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.360548 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 23:12:40.361569 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.361533 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:12:40.364318 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.364297 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.367250 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.367219 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 23:12:40.367353 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.367297 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pt5bp" Oct 29 23:12:40.369986 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.369969 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:40.370198 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:40.370175 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:12:40.372749 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.372728 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.383771 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.383721 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" event={"ID":"3a7e411141454709a32dcea46ca88fb7","Type":"ContainerStarted","Data":"fbe5b256b4f104384b21ffe6e16f9f413a15378154a8980529b4044551b908a4"} Oct 29 23:12:40.385006 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.384977 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" event={"ID":"563071327f3cc335e24937880997d7f4","Type":"ContainerStarted","Data":"eb5fcf7da1f60dc265c4cec8b0771d99a3915acbb6d6488ee5bb7fb65298f379"} Oct 29 23:12:40.407469 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.407447 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 23:12:40.419668 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.419640 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-kubelet\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.419929 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.419893 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-slash\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.420073 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.420054 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.420227 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.420209 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34499222-f746-4e87-8dc8-54e6a25c75d6-ovn-node-metrics-cert\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.420367 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.420351 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56n47\" (UniqueName: \"kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47\") pod \"network-check-target-69jbt\" (UID: \"0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc\") " pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:40.420567 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.420464 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcec0d7f-0330-434c-824a-246928597652-cni-binary-copy\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.420735 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.420718 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-cni-bin\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.420880 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.420864 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm9rx\" (UniqueName: \"kubernetes.io/projected/2c3e5ae7-809c-475e-9623-d4d96e6da95c-kube-api-access-jm9rx\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.421017 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.420993 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-ovn\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.421247 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.421228 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-etc-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.421378 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.421352 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.421582 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.421565 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysctl-conf\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.421807 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.421746 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-run\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.421951 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.421932 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c3e5ae7-809c-475e-9623-d4d96e6da95c-tmp\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.422170 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.422153 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e09503-45e8-4e94-9a3b-a5d291a3990f-tmp-dir\") pod \"node-resolver-f456x\" (UID: \"76e09503-45e8-4e94-9a3b-a5d291a3990f\") " pod="openshift-dns/node-resolver-f456x" Oct 29 23:12:40.422390 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.422331 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-var-lib-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.422532 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.422515 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.422762 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.422701 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kw4s\" (UniqueName: \"kubernetes.io/projected/34499222-f746-4e87-8dc8-54e6a25c75d6-kube-api-access-4kw4s\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.422905 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.422890 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-cnibin\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.423072 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.423056 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-socket-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.423242 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.423225 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-node-log\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.423391 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.423375 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cnibin\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.423665 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.423639 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.423822 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.423806 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6170bf16-e81d-40fd-a4a0-449213b94d19-serviceca\") pod \"node-ca-tr7bl\" (UID: \"6170bf16-e81d-40fd-a4a0-449213b94d19\") " pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:12:40.424000 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.423984 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-multus-certs\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.424186 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.424167 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-modprobe-d\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.424375 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.424358 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-systemd\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.424547 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.424531 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-system-cni-dir\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.424794 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.424719 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysconfig\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.424927 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.424912 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-kubernetes\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.425064 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.425048 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-registration-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.426332 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.426280 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-device-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.427031 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.426514 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/76e09503-45e8-4e94-9a3b-a5d291a3990f-hosts-file\") pod \"node-resolver-f456x\" (UID: \"76e09503-45e8-4e94-9a3b-a5d291a3990f\") " pod="openshift-dns/node-resolver-f456x" Oct 29 23:12:40.427185 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427161 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/254e83a6-ab81-4b6b-b5ce-0538ccaede49-dbus\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:40.427310 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427198 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc84k\" (UniqueName: \"kubernetes.io/projected/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-kube-api-access-wc84k\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.427310 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427226 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-etc-kubernetes\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.427310 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427288 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr5kp\" (UniqueName: \"kubernetes.io/projected/76e09503-45e8-4e94-9a3b-a5d291a3990f-kube-api-access-vr5kp\") pod \"node-resolver-f456x\" (UID: \"76e09503-45e8-4e94-9a3b-a5d291a3990f\") " pod="openshift-dns/node-resolver-f456x" Oct 29 23:12:40.427500 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427349 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6cba05f-3979-445f-a517-da054562cc20-konnectivity-ca\") pod \"konnectivity-agent-klctw\" (UID: \"e6cba05f-3979-445f-a517-da054562cc20\") " pod="kube-system/konnectivity-agent-klctw" Oct 29 23:12:40.427500 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427382 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-socket-dir-parent\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.427500 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427417 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-cni-multus\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.427500 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427444 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-hostroot\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.427500 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427477 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.427701 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427517 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l78l\" (UniqueName: \"kubernetes.io/projected/f4a78746-396b-46fd-a42a-d819d866a6af-kube-api-access-2l78l\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.427701 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427578 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-log-socket\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.427701 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427617 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/254e83a6-ab81-4b6b-b5ce-0538ccaede49-kubelet-config\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:40.427701 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427655 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-k8s-cni-cncf-io\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.427701 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427694 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-systemd-units\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.427881 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427737 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-systemd\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.427881 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427761 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-cni-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.427881 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427808 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-cni-netd\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.427881 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427843 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6170bf16-e81d-40fd-a4a0-449213b94d19-host\") pod \"node-ca-tr7bl\" (UID: \"6170bf16-e81d-40fd-a4a0-449213b94d19\") " pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:12:40.427881 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427874 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-netns\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.428082 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427925 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-sys\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.428082 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427942 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b21e162-a2f3-4505-9dff-34a47e20d3e6-host-slash\") pod \"iptables-alerter-twqrq\" (UID: \"1b21e162-a2f3-4505-9dff-34a47e20d3e6\") " pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:12:40.428082 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427958 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-ovnkube-config\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.428082 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427973 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-env-overrides\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.428082 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.427995 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.428082 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428016 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysctl-d\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.428082 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428032 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-tuned\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.428082 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428047 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:40.428082 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428070 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvnmz\" (UniqueName: \"kubernetes.io/projected/1b21e162-a2f3-4505-9dff-34a47e20d3e6-kube-api-access-hvnmz\") pod \"iptables-alerter-twqrq\" (UID: \"1b21e162-a2f3-4505-9dff-34a47e20d3e6\") " pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:12:40.428477 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428118 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-cni-bin\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.428477 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428150 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.428477 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428177 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bcec0d7f-0330-434c-824a-246928597652-multus-daemon-config\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.428477 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428206 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1b21e162-a2f3-4505-9dff-34a47e20d3e6-iptables-alerter-script\") pod \"iptables-alerter-twqrq\" (UID: \"1b21e162-a2f3-4505-9dff-34a47e20d3e6\") " pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:12:40.428477 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428233 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-run-netns\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.428477 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428263 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-ovnkube-script-lib\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.428477 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428292 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:40.428477 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428317 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-kubelet\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.428477 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428366 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-lib-modules\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.428477 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428394 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-host\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.428477 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428422 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8zs9\" (UniqueName: \"kubernetes.io/projected/54576e13-0668-4255-952b-46b2bf5a279c-kube-api-access-j8zs9\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:40.428477 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428449 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-sys-fs\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.428477 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428476 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6cba05f-3979-445f-a517-da054562cc20-agent-certs\") pod \"konnectivity-agent-klctw\" (UID: \"e6cba05f-3979-445f-a517-da054562cc20\") " pod="kube-system/konnectivity-agent-klctw" Oct 29 23:12:40.429084 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428527 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv2xt\" (UniqueName: \"kubernetes.io/projected/6170bf16-e81d-40fd-a4a0-449213b94d19-kube-api-access-zv2xt\") pod \"node-ca-tr7bl\" (UID: \"6170bf16-e81d-40fd-a4a0-449213b94d19\") " pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:12:40.429084 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428729 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-system-cni-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.429084 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428772 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-conf-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.429084 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428817 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.429084 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428854 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.429084 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428884 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg42k\" (UniqueName: \"kubernetes.io/projected/bcec0d7f-0330-434c-824a-246928597652-kube-api-access-mg42k\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.429084 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.428909 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-var-lib-kubelet\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.468068 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.468030 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 23:12:40.468343 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.468318 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 23:12:40.468343 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.468337 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 23:12:40.468697 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.468539 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 23:12:40.468697 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.468551 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 23:12:40.468697 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.468598 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 23:12:40.468697 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.468605 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pnrbh\"" Oct 29 23:12:40.468697 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.468608 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 23:12:40.468697 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.468706 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 23:12:40.468697 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.468718 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 23:12:40.469278 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.469157 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-t8tv9\"" Oct 29 23:12:40.469278 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.469206 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 23:12:40.469416 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.469399 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9w58g\"" Oct 29 23:12:40.469475 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.469429 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jb6zf\"" Oct 29 23:12:40.469475 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.469454 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 23:12:40.469607 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.469561 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 23:12:40.469607 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.469565 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 23:12:40.469607 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.469583 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 23:12:40.469708 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.469625 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 23:12:40.469708 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.469627 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 23:12:40.469708 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.469637 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 23:12:40.483222 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.483151 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 23:12:40.483222 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.483176 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pgvzg\"" Oct 29 23:12:40.483222 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.483197 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8km5l\"" Oct 29 23:12:40.483222 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.483200 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7zkzr\"" Oct 29 23:12:40.483581 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.483279 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 23:12:40.529911 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.529875 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-socket-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.530114 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.529928 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-node-log\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.530114 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.529959 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cnibin\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.530114 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.529984 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.530114 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530004 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6170bf16-e81d-40fd-a4a0-449213b94d19-serviceca\") pod \"node-ca-tr7bl\" (UID: \"6170bf16-e81d-40fd-a4a0-449213b94d19\") " pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:12:40.530114 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530078 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-multus-certs\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.530402 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530119 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cnibin\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.530402 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530084 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-node-log\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.530402 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530122 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-modprobe-d\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.530402 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530157 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-multus-certs\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.530402 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530180 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.530402 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530186 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-systemd\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.530402 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530219 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-system-cni-dir\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.530402 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530080 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-socket-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.530402 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530234 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-systemd\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.530402 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530249 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysconfig\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.530402 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530268 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-system-cni-dir\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.530402 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530243 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-modprobe-d\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.530402 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530298 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysconfig\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.531021 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530619 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6170bf16-e81d-40fd-a4a0-449213b94d19-serviceca\") pod \"node-ca-tr7bl\" (UID: \"6170bf16-e81d-40fd-a4a0-449213b94d19\") " pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:12:40.531021 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530767 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-kubernetes\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.531021 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530808 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-registration-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.531021 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530838 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-device-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.531021 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530863 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/76e09503-45e8-4e94-9a3b-a5d291a3990f-hosts-file\") pod \"node-resolver-f456x\" (UID: \"76e09503-45e8-4e94-9a3b-a5d291a3990f\") " pod="openshift-dns/node-resolver-f456x" Oct 29 23:12:40.531021 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530874 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-registration-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.531021 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530886 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/254e83a6-ab81-4b6b-b5ce-0538ccaede49-dbus\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:40.531021 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530899 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-device-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.531021 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530813 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-kubernetes\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.531021 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530913 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc84k\" (UniqueName: \"kubernetes.io/projected/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-kube-api-access-wc84k\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.531021 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530938 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/76e09503-45e8-4e94-9a3b-a5d291a3990f-hosts-file\") pod \"node-resolver-f456x\" (UID: \"76e09503-45e8-4e94-9a3b-a5d291a3990f\") " pod="openshift-dns/node-resolver-f456x" Oct 29 23:12:40.531021 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530944 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-etc-kubernetes\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.531021 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530965 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-etc-kubernetes\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.531021 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.530977 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr5kp\" (UniqueName: \"kubernetes.io/projected/76e09503-45e8-4e94-9a3b-a5d291a3990f-kube-api-access-vr5kp\") pod \"node-resolver-f456x\" (UID: \"76e09503-45e8-4e94-9a3b-a5d291a3990f\") " pod="openshift-dns/node-resolver-f456x" Oct 29 23:12:40.531021 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531011 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6cba05f-3979-445f-a517-da054562cc20-konnectivity-ca\") pod \"konnectivity-agent-klctw\" (UID: \"e6cba05f-3979-445f-a517-da054562cc20\") " pod="kube-system/konnectivity-agent-klctw" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531035 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-socket-dir-parent\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531062 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-cni-multus\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531087 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-hostroot\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531095 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/254e83a6-ab81-4b6b-b5ce-0538ccaede49-dbus\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531112 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531122 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-socket-dir-parent\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531138 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l78l\" (UniqueName: \"kubernetes.io/projected/f4a78746-396b-46fd-a42a-d819d866a6af-kube-api-access-2l78l\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531148 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-cni-multus\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531165 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-log-socket\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531180 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-hostroot\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531189 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531192 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/254e83a6-ab81-4b6b-b5ce-0538ccaede49-kubelet-config\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531217 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-k8s-cni-cncf-io\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531220 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-log-socket\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531238 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/254e83a6-ab81-4b6b-b5ce-0538ccaede49-kubelet-config\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531246 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-systemd-units\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.531684 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531261 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-k8s-cni-cncf-io\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531266 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-systemd\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531283 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-systemd-units\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531289 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-systemd\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531306 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-cni-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531331 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-cni-netd\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531355 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6170bf16-e81d-40fd-a4a0-449213b94d19-host\") pod \"node-ca-tr7bl\" (UID: \"6170bf16-e81d-40fd-a4a0-449213b94d19\") " pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531379 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-netns\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531403 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-sys\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531414 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6170bf16-e81d-40fd-a4a0-449213b94d19-host\") pod \"node-ca-tr7bl\" (UID: \"6170bf16-e81d-40fd-a4a0-449213b94d19\") " pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531429 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b21e162-a2f3-4505-9dff-34a47e20d3e6-host-slash\") pod \"iptables-alerter-twqrq\" (UID: \"1b21e162-a2f3-4505-9dff-34a47e20d3e6\") " pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531405 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-cni-netd\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531405 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-cni-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531457 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-ovnkube-config\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531442 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-netns\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531457 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-sys\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531515 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b21e162-a2f3-4505-9dff-34a47e20d3e6-host-slash\") pod \"iptables-alerter-twqrq\" (UID: \"1b21e162-a2f3-4505-9dff-34a47e20d3e6\") " pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531518 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-env-overrides\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.532504 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531549 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531567 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysctl-d\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531583 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-tuned\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531617 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531636 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvnmz\" (UniqueName: \"kubernetes.io/projected/1b21e162-a2f3-4505-9dff-34a47e20d3e6-kube-api-access-hvnmz\") pod \"iptables-alerter-twqrq\" (UID: \"1b21e162-a2f3-4505-9dff-34a47e20d3e6\") " pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531648 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6cba05f-3979-445f-a517-da054562cc20-konnectivity-ca\") pod \"konnectivity-agent-klctw\" (UID: \"e6cba05f-3979-445f-a517-da054562cc20\") " pod="kube-system/konnectivity-agent-klctw" Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531659 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-cni-bin\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531685 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531711 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bcec0d7f-0330-434c-824a-246928597652-multus-daemon-config\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531738 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1b21e162-a2f3-4505-9dff-34a47e20d3e6-iptables-alerter-script\") pod \"iptables-alerter-twqrq\" (UID: \"1b21e162-a2f3-4505-9dff-34a47e20d3e6\") " pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:40.531754 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531759 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysctl-d\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531788 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-run-netns\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531811 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-cni-bin\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:40.531848 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs podName:54576e13-0668-4255-952b-46b2bf5a279c nodeName:}" failed. No retries permitted until 2026-10-29 23:12:41.031806966 +0000 UTC m=+3.103335516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs") pod "network-metrics-daemon-xwn99" (UID: "54576e13-0668-4255-952b-46b2bf5a279c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531871 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-ovnkube-script-lib\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531904 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:40.533164 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531931 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-kubelet\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531957 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-lib-modules\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531978 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-host\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532000 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-env-overrides\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532073 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-ovnkube-config\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532078 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-host\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532125 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-kubelet\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532007 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8zs9\" (UniqueName: \"kubernetes.io/projected/54576e13-0668-4255-952b-46b2bf5a279c-kube-api-access-j8zs9\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531979 2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532192 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-lib-modules\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532199 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-sys-fs\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.531845 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-run-netns\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532159 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-sys-fs\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532292 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532338 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6cba05f-3979-445f-a517-da054562cc20-agent-certs\") pod \"konnectivity-agent-klctw\" (UID: \"e6cba05f-3979-445f-a517-da054562cc20\") " pod="kube-system/konnectivity-agent-klctw" Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532366 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:40.532370 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 23:12:40.534066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532405 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv2xt\" (UniqueName: \"kubernetes.io/projected/6170bf16-e81d-40fd-a4a0-449213b94d19-kube-api-access-zv2xt\") pod \"node-ca-tr7bl\" (UID: \"6170bf16-e81d-40fd-a4a0-449213b94d19\") " pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:12:40.534987 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:40.532519 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret podName:254e83a6-ab81-4b6b-b5ce-0538ccaede49 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:41.03250157 +0000 UTC m=+3.104030121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret") pod "global-pull-secret-syncer-xcszc" (UID: "254e83a6-ab81-4b6b-b5ce-0538ccaede49") : object "kube-system"/"original-pull-secret" not registered Oct 29 23:12:40.534987 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532588 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-system-cni-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.534987 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532629 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-conf-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.534987 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532653 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.534987 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532676 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.534987 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532676 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-system-cni-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.534987 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532694 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg42k\" (UniqueName: \"kubernetes.io/projected/bcec0d7f-0330-434c-824a-246928597652-kube-api-access-mg42k\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.534987 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532698 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-conf-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.534987 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532713 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-var-lib-kubelet\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.534987 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532749 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-kubelet\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.534987 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532756 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.534987 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532773 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-slash\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.534987 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532795 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-var-lib-kubelet\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.534987 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532800 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.534987 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532823 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34499222-f746-4e87-8dc8-54e6a25c75d6-ovn-node-metrics-cert\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.534987 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532845 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56n47\" (UniqueName: \"kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47\") pod \"network-check-target-69jbt\" (UID: \"0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc\") " pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532866 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcec0d7f-0330-434c-824a-246928597652-cni-binary-copy\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532888 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-cni-bin\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532911 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm9rx\" (UniqueName: \"kubernetes.io/projected/2c3e5ae7-809c-475e-9623-d4d96e6da95c-kube-api-access-jm9rx\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532934 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-ovn\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532956 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-etc-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.532978 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533001 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysctl-conf\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533022 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-run\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533044 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c3e5ae7-809c-475e-9623-d4d96e6da95c-tmp\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533066 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e09503-45e8-4e94-9a3b-a5d291a3990f-tmp-dir\") pod \"node-resolver-f456x\" (UID: \"76e09503-45e8-4e94-9a3b-a5d291a3990f\") " pod="openshift-dns/node-resolver-f456x" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533090 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-var-lib-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533111 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533135 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kw4s\" (UniqueName: \"kubernetes.io/projected/34499222-f746-4e87-8dc8-54e6a25c75d6-kube-api-access-4kw4s\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533156 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-cnibin\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533251 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-cnibin\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533277 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bcec0d7f-0330-434c-824a-246928597652-multus-daemon-config\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533349 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-slash\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.535843 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533396 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-kubelet\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.536754 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533436 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1b21e162-a2f3-4505-9dff-34a47e20d3e6-iptables-alerter-script\") pod \"iptables-alerter-twqrq\" (UID: \"1b21e162-a2f3-4505-9dff-34a47e20d3e6\") " pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:12:40.536754 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533446 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.536754 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533520 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-ovnkube-script-lib\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.536754 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533517 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-run\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.536754 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533595 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysctl-conf\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.536754 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533620 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-ovn\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.536754 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533664 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.536754 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533674 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-cni-bin\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.536754 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533728 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-etc-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.536754 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533778 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.536754 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533822 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.536754 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533864 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-var-lib-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.536754 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.533890 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e09503-45e8-4e94-9a3b-a5d291a3990f-tmp-dir\") pod \"node-resolver-f456x\" (UID: \"76e09503-45e8-4e94-9a3b-a5d291a3990f\") " pod="openshift-dns/node-resolver-f456x" Oct 29 23:12:40.536754 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.534368 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcec0d7f-0330-434c-824a-246928597652-cni-binary-copy\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.536754 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.536321 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-tuned\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.536754 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.536664 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6cba05f-3979-445f-a517-da054562cc20-agent-certs\") pod \"konnectivity-agent-klctw\" (UID: \"e6cba05f-3979-445f-a517-da054562cc20\") " pod="kube-system/konnectivity-agent-klctw" Oct 29 23:12:40.539542 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.539480 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c3e5ae7-809c-475e-9623-d4d96e6da95c-tmp\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.541116 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.541091 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34499222-f746-4e87-8dc8-54e6a25c75d6-ovn-node-metrics-cert\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.553808 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:40.553782 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 23:12:40.553930 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:40.553822 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 23:12:40.553930 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:40.553839 2571 projected.go:196] Error preparing data for projected volume kube-api-access-56n47 for pod openshift-network-diagnostics/network-check-target-69jbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 23:12:40.553930 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:40.553911 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47 podName:0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc nodeName:}" failed. No retries permitted until 2026-10-29 23:12:41.05388429 +0000 UTC m=+3.125412829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-56n47" (UniqueName: "kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47") pod "network-check-target-69jbt" (UID: "0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 23:12:40.556036 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.556005 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l78l\" (UniqueName: \"kubernetes.io/projected/f4a78746-396b-46fd-a42a-d819d866a6af-kube-api-access-2l78l\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.556502 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.556477 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc84k\" (UniqueName: \"kubernetes.io/projected/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-kube-api-access-wc84k\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.556821 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.556805 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv2xt\" (UniqueName: \"kubernetes.io/projected/6170bf16-e81d-40fd-a4a0-449213b94d19-kube-api-access-zv2xt\") pod \"node-ca-tr7bl\" (UID: \"6170bf16-e81d-40fd-a4a0-449213b94d19\") " pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:12:40.557984 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.557957 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kw4s\" (UniqueName: \"kubernetes.io/projected/34499222-f746-4e87-8dc8-54e6a25c75d6-kube-api-access-4kw4s\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.563981 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.563950 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg42k\" (UniqueName: \"kubernetes.io/projected/bcec0d7f-0330-434c-824a-246928597652-kube-api-access-mg42k\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.564067 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.563986 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvnmz\" (UniqueName: \"kubernetes.io/projected/1b21e162-a2f3-4505-9dff-34a47e20d3e6-kube-api-access-hvnmz\") pod \"iptables-alerter-twqrq\" (UID: \"1b21e162-a2f3-4505-9dff-34a47e20d3e6\") " pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:12:40.564377 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.564355 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr5kp\" (UniqueName: \"kubernetes.io/projected/76e09503-45e8-4e94-9a3b-a5d291a3990f-kube-api-access-vr5kp\") pod \"node-resolver-f456x\" (UID: \"76e09503-45e8-4e94-9a3b-a5d291a3990f\") " pod="openshift-dns/node-resolver-f456x" Oct 29 23:12:40.564447 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.564425 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm9rx\" (UniqueName: \"kubernetes.io/projected/2c3e5ae7-809c-475e-9623-d4d96e6da95c-kube-api-access-jm9rx\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.564899 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.564872 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8zs9\" (UniqueName: \"kubernetes.io/projected/54576e13-0668-4255-952b-46b2bf5a279c-kube-api-access-j8zs9\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:40.664632 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.664589 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:12:40.676424 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:40.676380 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a78746_396b_46fd_a42a_d819d866a6af.slice/crio-a60a7805f22a1f6866107dc1e940782d727d1953974dc2eb1f1b264591047daf WatchSource:0}: Error finding container a60a7805f22a1f6866107dc1e940782d727d1953974dc2eb1f1b264591047daf: Status 404 returned error can't find the container with id a60a7805f22a1f6866107dc1e940782d727d1953974dc2eb1f1b264591047daf Oct 29 23:12:40.681841 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.681628 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f456x" Oct 29 23:12:40.687301 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.687271 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:12:40.688441 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.688414 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:12:40.694331 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:40.694122 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e09503_45e8_4e94_9a3b_a5d291a3990f.slice/crio-7b59a50e31096da6b08d11a397229750794b1e6af3c9959372a89b4288cd992d WatchSource:0}: Error finding container 7b59a50e31096da6b08d11a397229750794b1e6af3c9959372a89b4288cd992d: Status 404 returned error can't find the container with id 7b59a50e31096da6b08d11a397229750794b1e6af3c9959372a89b4288cd992d Oct 29 23:12:40.697033 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.696754 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-klctw" Oct 29 23:12:40.700804 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:40.700777 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85fb3a66_9c1e_4acc_bd3f_82b115c2b4f9.slice/crio-56b460e0df4627dca42311dd173e6288fa2566a4c7b3ad1cfb32b6aa348178c4 WatchSource:0}: Error finding container 56b460e0df4627dca42311dd173e6288fa2566a4c7b3ad1cfb32b6aa348178c4: Status 404 returned error can't find the container with id 56b460e0df4627dca42311dd173e6288fa2566a4c7b3ad1cfb32b6aa348178c4 Oct 29 23:12:40.702835 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:40.702762 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b21e162_a2f3_4505_9dff_34a47e20d3e6.slice/crio-daaf958f300d0ddc0f2ccd8db050c5938b2f8f5eefc6781a84ee4100ad30380a WatchSource:0}: Error finding container daaf958f300d0ddc0f2ccd8db050c5938b2f8f5eefc6781a84ee4100ad30380a: Status 404 returned error can't find the container with id daaf958f300d0ddc0f2ccd8db050c5938b2f8f5eefc6781a84ee4100ad30380a Oct 29 23:12:40.709581 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:40.709554 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6cba05f_3979_445f_a517_da054562cc20.slice/crio-01c61799f0cefae338425ed766986726baf5fdb2c6f21da29ea96e925f83766c WatchSource:0}: Error finding container 01c61799f0cefae338425ed766986726baf5fdb2c6f21da29ea96e925f83766c: Status 404 returned error can't find the container with id 01c61799f0cefae338425ed766986726baf5fdb2c6f21da29ea96e925f83766c Oct 29 23:12:40.716728 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.716702 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:12:40.719398 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.719375 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ltlxg" Oct 29 23:12:40.723810 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.723783 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:12:40.726353 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.726329 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:12:40.733213 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:40.733131 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6170bf16_e81d_40fd_a4a0_449213b94d19.slice/crio-1e95efb7515cff6c8da04318ad7bd15f28b23b54e978ce0d7ffba071236296d0 WatchSource:0}: Error finding container 1e95efb7515cff6c8da04318ad7bd15f28b23b54e978ce0d7ffba071236296d0: Status 404 returned error can't find the container with id 1e95efb7515cff6c8da04318ad7bd15f28b23b54e978ce0d7ffba071236296d0 Oct 29 23:12:40.734008 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:40.733677 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcec0d7f_0330_434c_824a_246928597652.slice/crio-2e016aca5752d2009a16b9120e9fecfd95d1f88efd55a16131bacab7721de5f6 WatchSource:0}: Error finding container 2e016aca5752d2009a16b9120e9fecfd95d1f88efd55a16131bacab7721de5f6: Status 404 returned error can't find the container with id 2e016aca5752d2009a16b9120e9fecfd95d1f88efd55a16131bacab7721de5f6 Oct 29 23:12:40.738637 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:40.738616 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34499222_f746_4e87_8dc8_54e6a25c75d6.slice/crio-ebc69002251bf669a9de3554274ef6e438737d19545670996eb1105f888e93a7 WatchSource:0}: Error finding container ebc69002251bf669a9de3554274ef6e438737d19545670996eb1105f888e93a7: Status 404 returned error can't find the container with id ebc69002251bf669a9de3554274ef6e438737d19545670996eb1105f888e93a7 Oct 29 23:12:40.741835 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:12:40.741806 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3e5ae7_809c_475e_9623_d4d96e6da95c.slice/crio-5200d6078cef59b0bdf74406bf3005a7051b9da4d20b390b67ad96d6fd35e4cb WatchSource:0}: Error finding container 5200d6078cef59b0bdf74406bf3005a7051b9da4d20b390b67ad96d6fd35e4cb: Status 404 returned error can't find the container with id 5200d6078cef59b0bdf74406bf3005a7051b9da4d20b390b67ad96d6fd35e4cb Oct 29 23:12:40.760363 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:40.760341 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 23:12:41.037766 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.037722 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:41.037766 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.037781 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:41.037993 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:41.037909 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 23:12:41.037993 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:41.037968 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret podName:254e83a6-ab81-4b6b-b5ce-0538ccaede49 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:42.037954676 +0000 UTC m=+4.109483192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret") pod "global-pull-secret-syncer-xcszc" (UID: "254e83a6-ab81-4b6b-b5ce-0538ccaede49") : object "kube-system"/"original-pull-secret" not registered Oct 29 23:12:41.038395 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:41.038372 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 23:12:41.038516 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:41.038432 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs podName:54576e13-0668-4255-952b-46b2bf5a279c nodeName:}" failed. No retries permitted until 2026-10-29 23:12:42.038414259 +0000 UTC m=+4.109942790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs") pod "network-metrics-daemon-xwn99" (UID: "54576e13-0668-4255-952b-46b2bf5a279c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 23:12:41.138365 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.138323 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56n47\" (UniqueName: \"kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47\") pod \"network-check-target-69jbt\" (UID: \"0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc\") " pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:41.138542 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:41.138509 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 23:12:41.138620 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:41.138548 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 23:12:41.138620 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:41.138563 2571 projected.go:196] Error preparing data for projected volume kube-api-access-56n47 for pod openshift-network-diagnostics/network-check-target-69jbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 23:12:41.138739 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:41.138625 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47 podName:0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc nodeName:}" failed. No retries permitted until 2026-10-29 23:12:42.138605515 +0000 UTC m=+4.210134047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-56n47" (UniqueName: "kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47") pod "network-check-target-69jbt" (UID: "0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 23:12:41.323654 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.323557 2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 23:12:41.371867 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.371730 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 23:07:40 +0000 UTC" deadline="2028-04-01 07:24:35.620648203 +0000 UTC" Oct 29 23:12:41.371867 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.371799 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12464h11m54.248853807s" Oct 29 23:12:41.379377 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.378793 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:41.379377 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:41.378952 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:12:41.394522 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.394316 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ccm62" event={"ID":"2c3e5ae7-809c-475e-9623-d4d96e6da95c","Type":"ContainerStarted","Data":"5200d6078cef59b0bdf74406bf3005a7051b9da4d20b390b67ad96d6fd35e4cb"} Oct 29 23:12:41.401024 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.400901 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" event={"ID":"34499222-f746-4e87-8dc8-54e6a25c75d6","Type":"ContainerStarted","Data":"ebc69002251bf669a9de3554274ef6e438737d19545670996eb1105f888e93a7"} Oct 29 23:12:41.422919 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.422879 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-klctw" event={"ID":"e6cba05f-3979-445f-a517-da054562cc20","Type":"ContainerStarted","Data":"01c61799f0cefae338425ed766986726baf5fdb2c6f21da29ea96e925f83766c"} Oct 29 23:12:41.436589 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.436473 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-twqrq" event={"ID":"1b21e162-a2f3-4505-9dff-34a47e20d3e6","Type":"ContainerStarted","Data":"daaf958f300d0ddc0f2ccd8db050c5938b2f8f5eefc6781a84ee4100ad30380a"} Oct 29 23:12:41.451753 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.451709 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" event={"ID":"f4a78746-396b-46fd-a42a-d819d866a6af","Type":"ContainerStarted","Data":"a60a7805f22a1f6866107dc1e940782d727d1953974dc2eb1f1b264591047daf"} Oct 29 23:12:41.463475 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.463419 2571 generic.go:356] "Generic (PLEG): container finished" podID="563071327f3cc335e24937880997d7f4" containerID="f8257adbd933d9107b5ce29dff2f2c66a2377748ffe4d75532599bb7fe7d6ac5" exitCode=0 Oct 29 23:12:41.463655 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.463523 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" event={"ID":"563071327f3cc335e24937880997d7f4","Type":"ContainerDied","Data":"f8257adbd933d9107b5ce29dff2f2c66a2377748ffe4d75532599bb7fe7d6ac5"} Oct 29 23:12:41.473097 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.473059 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" event={"ID":"3a7e411141454709a32dcea46ca88fb7","Type":"ContainerStarted","Data":"49c8656f64c9cbedc34216c69440d1457dcd3607976a0547d4bd52b030bcf1ba"} Oct 29 23:12:41.477118 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.477040 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ltlxg" event={"ID":"bcec0d7f-0330-434c-824a-246928597652","Type":"ContainerStarted","Data":"2e016aca5752d2009a16b9120e9fecfd95d1f88efd55a16131bacab7721de5f6"} Oct 29 23:12:41.491600 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.491492 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tr7bl" event={"ID":"6170bf16-e81d-40fd-a4a0-449213b94d19","Type":"ContainerStarted","Data":"1e95efb7515cff6c8da04318ad7bd15f28b23b54e978ce0d7ffba071236296d0"} Oct 29 23:12:41.495770 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.495736 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn78d" event={"ID":"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9","Type":"ContainerStarted","Data":"56b460e0df4627dca42311dd173e6288fa2566a4c7b3ad1cfb32b6aa348178c4"} Oct 29 23:12:41.500965 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:41.500929 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f456x" event={"ID":"76e09503-45e8-4e94-9a3b-a5d291a3990f","Type":"ContainerStarted","Data":"7b59a50e31096da6b08d11a397229750794b1e6af3c9959372a89b4288cd992d"} Oct 29 23:12:42.046870 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:42.045945 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:42.046870 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:42.046003 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:42.046870 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:42.046178 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 23:12:42.046870 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:42.046260 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret podName:254e83a6-ab81-4b6b-b5ce-0538ccaede49 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:44.046227033 +0000 UTC m=+6.117755555 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret") pod "global-pull-secret-syncer-xcszc" (UID: "254e83a6-ab81-4b6b-b5ce-0538ccaede49") : object "kube-system"/"original-pull-secret" not registered Oct 29 23:12:42.046870 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:42.046718 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 23:12:42.046870 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:42.046777 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs podName:54576e13-0668-4255-952b-46b2bf5a279c nodeName:}" failed. No retries permitted until 2026-10-29 23:12:44.04675881 +0000 UTC m=+6.118287349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs") pod "network-metrics-daemon-xwn99" (UID: "54576e13-0668-4255-952b-46b2bf5a279c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 23:12:42.146902 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:42.146860 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56n47\" (UniqueName: \"kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47\") pod \"network-check-target-69jbt\" (UID: \"0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc\") " pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:42.147102 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:42.147065 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 23:12:42.147160 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:42.147102 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 23:12:42.147160 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:42.147121 2571 projected.go:196] Error preparing data for projected volume kube-api-access-56n47 for pod openshift-network-diagnostics/network-check-target-69jbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 23:12:42.147267 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:42.147190 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47 podName:0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc nodeName:}" failed. No retries permitted until 2026-10-29 23:12:44.147167893 +0000 UTC m=+6.218696973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-56n47" (UniqueName: "kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47") pod "network-check-target-69jbt" (UID: "0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 23:12:42.372894 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:42.372752 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 23:07:40 +0000 UTC" deadline="2028-05-26 22:29:47.219702918 +0000 UTC" Oct 29 23:12:42.372894 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:42.372789 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13799h17m4.846918818s" Oct 29 23:12:42.384974 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:42.383949 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:42.384974 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:42.384098 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:12:42.384974 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:42.384522 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:42.384974 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:42.384674 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:12:42.520636 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:42.520280 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" event={"ID":"563071327f3cc335e24937880997d7f4","Type":"ContainerStarted","Data":"aca5f43ea6f0601cf0150b76a5ac8c71a9ee7da51d5a99f81d78b856b605a295"} Oct 29 23:12:42.569003 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:42.568934 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" podStartSLOduration=2.568912883 podStartE2EDuration="2.568912883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:41.563595706 +0000 UTC m=+3.635124246" watchObservedRunningTime="2026-10-29 23:12:42.568912883 +0000 UTC m=+4.640441422" Oct 29 23:12:43.379410 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:43.378728 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:43.379410 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:43.378883 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:12:44.063126 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:44.063087 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:44.063290 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:44.063142 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:44.063290 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:44.063263 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 23:12:44.063402 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:44.063326 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret podName:254e83a6-ab81-4b6b-b5ce-0538ccaede49 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:48.063308884 +0000 UTC m=+10.134837421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret") pod "global-pull-secret-syncer-xcszc" (UID: "254e83a6-ab81-4b6b-b5ce-0538ccaede49") : object "kube-system"/"original-pull-secret" not registered Oct 29 23:12:44.063745 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:44.063726 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 23:12:44.063822 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:44.063781 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs podName:54576e13-0668-4255-952b-46b2bf5a279c nodeName:}" failed. No retries permitted until 2026-10-29 23:12:48.063766926 +0000 UTC m=+10.135295454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs") pod "network-metrics-daemon-xwn99" (UID: "54576e13-0668-4255-952b-46b2bf5a279c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 23:12:44.163846 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:44.163493 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56n47\" (UniqueName: \"kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47\") pod \"network-check-target-69jbt\" (UID: \"0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc\") " pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:44.163846 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:44.163669 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 23:12:44.163846 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:44.163692 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 23:12:44.163846 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:44.163706 2571 projected.go:196] Error preparing data for projected volume kube-api-access-56n47 for pod openshift-network-diagnostics/network-check-target-69jbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 23:12:44.163846 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:44.163772 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47 podName:0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc nodeName:}" failed. No retries permitted until 2026-10-29 23:12:48.163753937 +0000 UTC m=+10.235282466 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-56n47" (UniqueName: "kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47") pod "network-check-target-69jbt" (UID: "0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 23:12:44.381963 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:44.381927 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:44.382389 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:44.382069 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:12:44.382535 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:44.382502 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:44.382633 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:44.382597 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:12:44.538940 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:44.538185 2571 generic.go:356] "Generic (PLEG): container finished" podID="85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9" containerID="c84d64b2d105cb404db4f430ab67ebd5325a98803d6d58da66648423301d346a" exitCode=0 Oct 29 23:12:44.538940 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:44.538232 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn78d" event={"ID":"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9","Type":"ContainerDied","Data":"c84d64b2d105cb404db4f430ab67ebd5325a98803d6d58da66648423301d346a"} Oct 29 23:12:44.615596 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:44.615018 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" podStartSLOduration=4.614998078 podStartE2EDuration="4.614998078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:42.569900685 +0000 UTC m=+4.641429225" watchObservedRunningTime="2026-10-29 23:12:44.614998078 +0000 UTC m=+6.686526615" Oct 29 23:12:45.379292 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:45.379253 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:45.379517 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:45.379396 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:12:45.547294 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:45.547250 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-klctw" event={"ID":"e6cba05f-3979-445f-a517-da054562cc20","Type":"ContainerStarted","Data":"9379bc9e000c2ea10306ac69a2cb43c935c430e52047a990259b13075eb64e17"} Oct 29 23:12:46.381326 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:46.380579 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:46.381326 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:46.380634 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:46.381326 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:46.380741 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:12:46.381326 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:46.380876 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:12:46.552399 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:46.552363 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" event={"ID":"f4a78746-396b-46fd-a42a-d819d866a6af","Type":"ContainerStarted","Data":"e26390144664b99ca7e5a619c56efb7475b444f8e64b162df84c38a08c51e130"} Oct 29 23:12:46.556150 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:46.556052 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tr7bl" event={"ID":"6170bf16-e81d-40fd-a4a0-449213b94d19","Type":"ContainerStarted","Data":"214d2da8b341261b71e06298dc2257a9926668139ca42ba6fd454e477b941a84"} Oct 29 23:12:46.596469 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:46.596387 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-klctw" podStartSLOduration=4.818459842 podStartE2EDuration="8.596365119s" totalImagesPullingTime="3.777905277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:45.589965896 +0000 UTC m=+7.661494434" watchObservedRunningTime="2026-10-29 23:12:46.596365119 +0000 UTC m=+8.667893656" Oct 29 23:12:47.043005 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:47.042970 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-klctw" Oct 29 23:12:47.043598 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:47.043577 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-klctw" Oct 29 23:12:47.092332 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:47.092079 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tr7bl" podStartSLOduration=3.933844412 podStartE2EDuration="9.09206332s" totalImagesPullingTime="5.158218908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:46.597066627 +0000 UTC m=+8.668595189" watchObservedRunningTime="2026-10-29 23:12:47.09206332 +0000 UTC m=+9.163591855" Oct 29 23:12:47.379311 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:47.378587 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:47.379311 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:47.378756 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:12:47.926777 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:47.926747 2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 23:12:48.098984 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:48.098904 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:48.098984 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:48.098962 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:48.099187 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:48.099100 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 23:12:48.099187 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:48.099164 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret podName:254e83a6-ab81-4b6b-b5ce-0538ccaede49 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:56.099144899 +0000 UTC m=+18.170673427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret") pod "global-pull-secret-syncer-xcszc" (UID: "254e83a6-ab81-4b6b-b5ce-0538ccaede49") : object "kube-system"/"original-pull-secret" not registered Oct 29 23:12:48.099581 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:48.099560 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 23:12:48.099682 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:48.099616 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs podName:54576e13-0668-4255-952b-46b2bf5a279c nodeName:}" failed. No retries permitted until 2026-10-29 23:12:56.099600391 +0000 UTC m=+18.171128947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs") pod "network-metrics-daemon-xwn99" (UID: "54576e13-0668-4255-952b-46b2bf5a279c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 23:12:48.200310 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:48.200271 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56n47\" (UniqueName: \"kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47\") pod \"network-check-target-69jbt\" (UID: \"0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc\") " pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:48.200480 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:48.200453 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 23:12:48.200559 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:48.200487 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 23:12:48.200559 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:48.200500 2571 projected.go:196] Error preparing data for projected volume kube-api-access-56n47 for pod openshift-network-diagnostics/network-check-target-69jbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 23:12:48.200631 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:48.200563 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47 podName:0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc nodeName:}" failed. No retries permitted until 2026-10-29 23:12:56.200544077 +0000 UTC m=+18.272072612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-56n47" (UniqueName: "kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47") pod "network-check-target-69jbt" (UID: "0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 23:12:48.380309 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:48.379869 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:48.380309 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:48.379863 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:48.380309 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:48.379969 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:12:48.380309 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:48.380048 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:12:48.387903 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:48.387800 2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T23:12:47.926770783Z","UUID":"53c77f10-5a19-4416-b9e9-bc252dfeb657","Handler":null,"Name":"","Endpoint":""} Oct 29 23:12:48.389647 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:48.389622 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 23:12:48.389783 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:48.389654 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 23:12:48.568117 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:48.568077 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" event={"ID":"f4a78746-396b-46fd-a42a-d819d866a6af","Type":"ContainerStarted","Data":"21005aeb06294199fb633594acf40d2318ecc917c6bd77b9edb9f2c552e1b011"} Oct 29 23:12:48.571813 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:48.570723 2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 23:12:48.571813 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:48.571700 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ltlxg" event={"ID":"bcec0d7f-0330-434c-824a-246928597652","Type":"ContainerStarted","Data":"6ef9234dca1714c75472933f0e44f1d892548a0e9f10b158b22f8cb376c1a1eb"} Oct 29 23:12:48.624863 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:48.624811 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ltlxg" podStartSLOduration=3.031845664 podStartE2EDuration="10.624794165s" totalImagesPullingTime="7.592948501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:48.624423676 +0000 UTC m=+10.695952213" watchObservedRunningTime="2026-10-29 23:12:48.624794165 +0000 UTC m=+10.696322702" Oct 29 23:12:49.379806 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:49.379560 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:49.380235 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:49.379897 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:12:49.577304 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:49.576642 2571 generic.go:356] "Generic (PLEG): container finished" podID="85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9" containerID="85c59069ff6058e7716172c696cebd8963c11d097af46cbae05ae76c8d3f669b" exitCode=0 Oct 29 23:12:49.577304 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:49.576728 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn78d" event={"ID":"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9","Type":"ContainerDied","Data":"85c59069ff6058e7716172c696cebd8963c11d097af46cbae05ae76c8d3f669b"} Oct 29 23:12:49.578999 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:49.578953 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f456x" event={"ID":"76e09503-45e8-4e94-9a3b-a5d291a3990f","Type":"ContainerStarted","Data":"d2cace653b916d892cda5f3f1b0046118dfc009552a446a3a6eafcc148ba2774"} Oct 29 23:12:49.582653 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:49.582578 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" event={"ID":"f4a78746-396b-46fd-a42a-d819d866a6af","Type":"ContainerStarted","Data":"8f498526c28017aab16d20702a784915ab1a30be3c17f20a96ef1fe8bf68a9bc"} Oct 29 23:12:49.664958 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:49.664907 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" podStartSLOduration=3.616269801 podStartE2EDuration="11.664887753s" totalImagesPullingTime="8.048617952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:49.664482415 +0000 UTC m=+11.736010952" watchObservedRunningTime="2026-10-29 23:12:49.664887753 +0000 UTC m=+11.736416302" Oct 29 23:12:49.705388 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:49.705319 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-f456x" podStartSLOduration=3.592617147 podStartE2EDuration="11.705302082s" totalImagesPullingTime="8.112684935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:49.70485552 +0000 UTC m=+11.776384056" watchObservedRunningTime="2026-10-29 23:12:49.705302082 +0000 UTC m=+11.776830618" Oct 29 23:12:49.914593 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:49.914410 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-klctw" Oct 29 23:12:49.914593 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:49.914530 2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 23:12:49.915410 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:49.915379 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-klctw" Oct 29 23:12:50.378842 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:50.378810 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:50.379007 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:50.378810 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:50.379007 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:50.378951 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:12:50.379007 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:50.378995 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:12:50.586808 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:50.586768 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-twqrq" event={"ID":"1b21e162-a2f3-4505-9dff-34a47e20d3e6","Type":"ContainerStarted","Data":"42caeaf0faef3877e05334e1da67408acc04a0c2b4eec3a737ebbbe48e8c51a5"} Oct 29 23:12:50.628365 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:50.628266 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-twqrq" podStartSLOduration=4.741578025 podStartE2EDuration="12.628251278s" totalImagesPullingTime="7.886673253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:50.627875652 +0000 UTC m=+12.699404199" watchObservedRunningTime="2026-10-29 23:12:50.628251278 +0000 UTC m=+12.699779814" Oct 29 23:12:51.379554 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:51.379453 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:51.379720 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:51.379601 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:12:51.591334 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:51.591297 2571 generic.go:356] "Generic (PLEG): container finished" podID="85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9" containerID="70a43c38036d94f3eed1db52bf5fa1871ad9588bdcd14d3497e79bd3b277693f" exitCode=0 Oct 29 23:12:51.592187 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:51.591373 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn78d" event={"ID":"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9","Type":"ContainerDied","Data":"70a43c38036d94f3eed1db52bf5fa1871ad9588bdcd14d3497e79bd3b277693f"} Oct 29 23:12:51.593816 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:51.593795 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ccm62" event={"ID":"2c3e5ae7-809c-475e-9623-d4d96e6da95c","Type":"ContainerStarted","Data":"0a4cbf6ed93f9e99405d090301fed0540b77a4e0b3bcadf3b1d7b07fd9101359"} Oct 29 23:12:52.378954 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:52.378913 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:52.378954 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:52.378954 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:52.379162 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:52.379054 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:12:52.379162 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:52.379150 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:12:52.597863 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:52.597827 2571 generic.go:356] "Generic (PLEG): container finished" podID="85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9" containerID="2b444d4c00171b3d9f84c366d156428e4bd1d642b5bbdba9f0c331af1bee168b" exitCode=0 Oct 29 23:12:52.598314 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:52.597910 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn78d" event={"ID":"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9","Type":"ContainerDied","Data":"2b444d4c00171b3d9f84c366d156428e4bd1d642b5bbdba9f0c331af1bee168b"} Oct 29 23:12:52.670286 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:52.670196 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ccm62" podStartSLOduration=3.372535124 podStartE2EDuration="13.670178109s" totalImagesPullingTime="10.297642985s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:51.694430256 +0000 UTC m=+13.765958819" watchObservedRunningTime="2026-10-29 23:12:52.670178109 +0000 UTC m=+14.741706644" Oct 29 23:12:53.379351 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:53.379326 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:53.379530 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:53.379430 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:12:54.378721 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:54.378686 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:54.379228 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:54.378731 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:54.379228 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:54.378854 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:12:54.379228 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:54.378984 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:12:54.604615 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:54.604577 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn78d" event={"ID":"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9","Type":"ContainerStarted","Data":"e7bebcffe8486b5a5930999d95310ee8ba35cde3dffbf832104a238a13fbc31d"} Oct 29 23:12:55.379393 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:55.379360 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:55.379802 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:55.379492 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:12:55.608709 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:55.608485 2571 generic.go:356] "Generic (PLEG): container finished" podID="85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9" containerID="e7bebcffe8486b5a5930999d95310ee8ba35cde3dffbf832104a238a13fbc31d" exitCode=0 Oct 29 23:12:55.608827 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:55.608550 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn78d" event={"ID":"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9","Type":"ContainerDied","Data":"e7bebcffe8486b5a5930999d95310ee8ba35cde3dffbf832104a238a13fbc31d"} Oct 29 23:12:55.610537 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:55.610516 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" event={"ID":"34499222-f746-4e87-8dc8-54e6a25c75d6","Type":"ContainerStarted","Data":"7fb1932cf71c0dc5a1d0f348cc7405143870a1cd8f04590ec4959179e62c80b5"} Oct 29 23:12:55.610626 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:55.610579 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" event={"ID":"34499222-f746-4e87-8dc8-54e6a25c75d6","Type":"ContainerStarted","Data":"13d86e880bd6d7833e8727471716f5b9b0e3b40b2bb454d8afbb3aaaced6ca5d"} Oct 29 23:12:56.161172 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:56.161138 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:56.161307 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:56.161186 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:56.161307 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:56.161287 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 23:12:56.161307 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:56.161299 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 23:12:56.161410 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:56.161358 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs podName:54576e13-0668-4255-952b-46b2bf5a279c nodeName:}" failed. No retries permitted until 2026-10-29 23:13:12.161339638 +0000 UTC m=+34.232868154 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs") pod "network-metrics-daemon-xwn99" (UID: "54576e13-0668-4255-952b-46b2bf5a279c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 23:12:56.161410 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:56.161378 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret podName:254e83a6-ab81-4b6b-b5ce-0538ccaede49 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:12.161369024 +0000 UTC m=+34.232897568 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret") pod "global-pull-secret-syncer-xcszc" (UID: "254e83a6-ab81-4b6b-b5ce-0538ccaede49") : object "kube-system"/"original-pull-secret" not registered Oct 29 23:12:56.261788 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:56.261751 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56n47\" (UniqueName: \"kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47\") pod \"network-check-target-69jbt\" (UID: \"0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc\") " pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:56.261958 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:56.261938 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 23:12:56.261991 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:56.261962 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 23:12:56.261991 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:56.261972 2571 projected.go:196] Error preparing data for projected volume kube-api-access-56n47 for pod openshift-network-diagnostics/network-check-target-69jbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 23:12:56.262095 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:56.262023 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47 podName:0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc nodeName:}" failed. No retries permitted until 2026-10-29 23:13:12.262010167 +0000 UTC m=+34.333538685 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-56n47" (UniqueName: "kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47") pod "network-check-target-69jbt" (UID: "0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 23:12:56.378815 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:56.378778 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:56.378815 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:56.378806 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:56.378996 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:56.378886 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:12:56.379124 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:56.379004 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:12:56.616115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:56.616084 2571 generic.go:356] "Generic (PLEG): container finished" podID="85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9" containerID="9fa2d4d135e7d77c279f5286ed10bc14e0dd4bb298aa4441ab35a61983e42c1e" exitCode=0 Oct 29 23:12:56.616646 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:56.616170 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn78d" event={"ID":"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9","Type":"ContainerDied","Data":"9fa2d4d135e7d77c279f5286ed10bc14e0dd4bb298aa4441ab35a61983e42c1e"} Oct 29 23:12:56.619217 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:56.619191 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" event={"ID":"34499222-f746-4e87-8dc8-54e6a25c75d6","Type":"ContainerStarted","Data":"e142b5df8c94afbb55c6d7c58a43d7bfba15e190cdebaf5b70febb8153658cd1"} Oct 29 23:12:56.619307 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:56.619250 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" event={"ID":"34499222-f746-4e87-8dc8-54e6a25c75d6","Type":"ContainerStarted","Data":"c1a1674a207460b6d1dcbb7f57724dd472f80c228b1f3fe248b7999a23dbee2b"} Oct 29 23:12:56.619307 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:56.619280 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" event={"ID":"34499222-f746-4e87-8dc8-54e6a25c75d6","Type":"ContainerStarted","Data":"6a2cffadaf856fe34a05cedff802034215d95a998e595b625be1ae267f7d7525"} Oct 29 23:12:56.619307 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:56.619307 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" event={"ID":"34499222-f746-4e87-8dc8-54e6a25c75d6","Type":"ContainerStarted","Data":"d330e930f4be0590ad966cae9e7d43d375fd5fdf217ecefbabbe3ede4d3ac80c"} Oct 29 23:12:57.379571 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:57.379520 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:57.379744 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:57.379646 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:12:57.624021 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:57.623986 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn78d" event={"ID":"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9","Type":"ContainerStarted","Data":"002f17240ac7840421251531383fb227e006838a6a16d5adabc78e6e733da332"} Oct 29 23:12:58.379662 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:58.379575 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:12:58.379863 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:58.379783 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:12:58.379863 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:58.379762 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:12:58.379863 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:58.379853 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:12:58.629059 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:58.629023 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" event={"ID":"34499222-f746-4e87-8dc8-54e6a25c75d6","Type":"ContainerStarted","Data":"d537ee981afe2612dca3b3f0e170b15c3b72d590c7f64660be08c7a18cf81035"} Oct 29 23:12:59.378941 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:12:59.378912 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:12:59.379104 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:12:59.379011 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:13:00.378860 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:00.378752 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:00.378860 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:00.378810 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:00.379335 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:00.378899 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:13:00.379335 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:00.379008 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:13:00.636867 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:00.636781 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" event={"ID":"34499222-f746-4e87-8dc8-54e6a25c75d6","Type":"ContainerStarted","Data":"e85d60308b439fa80fdab659d11aa68821b8dc1400c465b630d8d506a0218731"} Oct 29 23:13:00.637066 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:00.637035 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:00.637207 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:00.637115 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:00.652820 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:00.652795 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:00.653764 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:00.653746 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:00.728126 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:00.728072 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" podStartSLOduration=7.991430734 podStartE2EDuration="22.728058473s" totalImagesPullingTime="14.736627739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:00.727776834 +0000 UTC m=+22.799305369" watchObservedRunningTime="2026-10-29 23:13:00.728058473 +0000 UTC m=+22.799587008" Oct 29 23:13:00.728347 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:00.728329 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fn78d" podStartSLOduration=11.580504317 podStartE2EDuration="22.728322832s" totalImagesPullingTime="11.147818515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:57.705654755 +0000 UTC m=+19.777183291" watchObservedRunningTime="2026-10-29 23:13:00.728322832 +0000 UTC m=+22.799851367" Oct 29 23:13:01.379221 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:01.379186 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:01.379640 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:01.379297 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:13:01.639343 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:01.639266 2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 23:13:02.113472 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:02.113381 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xcszc"] Oct 29 23:13:02.113650 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:02.113521 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:02.113650 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:02.113614 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:13:02.115751 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:02.115727 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xwn99"] Oct 29 23:13:02.115874 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:02.115830 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:02.115955 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:02.115938 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:13:02.126983 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:02.126964 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-69jbt"] Oct 29 23:13:02.127056 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:02.127045 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:02.127132 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:02.127117 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:13:02.641387 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:02.641358 2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 23:13:04.379128 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:04.379051 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:04.379128 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:04.379077 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:04.379548 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:04.379051 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:04.379548 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:04.379182 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:13:04.379548 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:04.379294 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:13:04.379548 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:04.379384 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:13:06.378823 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:06.378729 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:06.378823 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:06.378754 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:06.379381 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:06.378831 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:06.379381 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:06.378855 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:13:06.379381 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:06.378977 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:13:06.379381 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:06.379052 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:13:07.353447 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:07.353375 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f456x_76e09503-45e8-4e94-9a3b-a5d291a3990f/dns-node-resolver/0.log" Oct 29 23:13:08.379613 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:08.379526 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:08.380081 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:08.379631 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:13:08.380081 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:08.379638 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:08.380081 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:08.379656 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:08.380081 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:08.379746 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:13:08.380081 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:08.379844 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:13:08.504334 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:08.504307 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tr7bl_6170bf16-e81d-40fd-a4a0-449213b94d19/node-ca/0.log" Oct 29 23:13:09.690980 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:09.690945 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-klctw_e6cba05f-3979-445f-a517-da054562cc20/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:09.881347 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:09.881256 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-klctw_e6cba05f-3979-445f-a517-da054562cc20/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:10.160676 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:10.160601 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:10.160859 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:10.160845 2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 23:13:10.181738 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:10.181713 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:10.379508 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:10.379471 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:10.379669 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:10.379525 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:10.379669 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:10.379533 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:10.379669 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:10.379616 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:13:10.379795 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:10.379695 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:13:10.379795 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:10.379775 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:13:12.166571 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:12.166494 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:12.166571 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:12.166531 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:12.166956 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:12.166630 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 23:13:12.166956 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:12.166634 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 23:13:12.166956 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:12.166676 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret podName:254e83a6-ab81-4b6b-b5ce-0538ccaede49 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:44.166663713 +0000 UTC m=+66.238192228 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret") pod "global-pull-secret-syncer-xcszc" (UID: "254e83a6-ab81-4b6b-b5ce-0538ccaede49") : object "kube-system"/"original-pull-secret" not registered Oct 29 23:13:12.166956 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:12.166690 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs podName:54576e13-0668-4255-952b-46b2bf5a279c nodeName:}" failed. No retries permitted until 2026-10-29 23:13:44.166683256 +0000 UTC m=+66.238211770 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs") pod "network-metrics-daemon-xwn99" (UID: "54576e13-0668-4255-952b-46b2bf5a279c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 23:13:12.266836 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:12.266797 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56n47\" (UniqueName: \"kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47\") pod \"network-check-target-69jbt\" (UID: \"0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc\") " pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:12.267002 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:12.266928 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 23:13:12.267002 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:12.266942 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 23:13:12.267002 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:12.266951 2571 projected.go:196] Error preparing data for projected volume kube-api-access-56n47 for pod openshift-network-diagnostics/network-check-target-69jbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 23:13:12.267002 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:12.267000 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47 podName:0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc nodeName:}" failed. No retries permitted until 2026-10-29 23:13:44.266987467 +0000 UTC m=+66.338515981 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-56n47" (UniqueName: "kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47") pod "network-check-target-69jbt" (UID: "0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 23:13:12.379216 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:12.379186 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:12.379373 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:12.379314 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:12.379373 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:12.379322 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:13:12.379453 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:12.379432 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:12.379522 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:12.379432 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:13:12.379562 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:12.379536 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:13:14.379440 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:14.379399 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:14.379844 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:14.379405 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:14.379844 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:14.379523 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:14.379844 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:14.379523 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:13:14.379844 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:14.379599 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:13:14.379844 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:14.379669 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:13:16.379078 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:16.379046 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:16.379535 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:16.379335 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:16.379535 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:16.379047 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:16.379677 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:16.379567 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:13:16.380302 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:16.380276 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:13:16.380505 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:16.380483 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:13:18.380028 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:18.380000 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:18.380491 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:18.380116 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:18.380491 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:18.380110 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:13:18.380491 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:18.380216 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:18.380491 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:18.380235 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:13:18.380491 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:18.380281 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:13:20.379200 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:20.379114 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:20.379200 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:20.379138 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:20.379628 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:20.379298 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:13:20.379628 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:20.379338 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:20.379628 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:20.379421 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:13:20.379628 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:20.379512 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:13:22.378928 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:22.378894 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:22.379413 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:22.378903 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:22.379413 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:22.379011 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:13:22.379413 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:22.378903 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:22.379413 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:22.379075 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:13:22.379413 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:22.379136 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:13:24.378840 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:24.378808 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:24.378840 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:24.378822 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:24.378840 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:24.378824 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:24.379399 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:24.378910 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xcszc" podUID="254e83a6-ab81-4b6b-b5ce-0538ccaede49" Oct 29 23:13:24.379399 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:24.379000 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69jbt" podUID="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" Oct 29 23:13:24.379399 ip-10-0-133-100 kubenswrapper[2571]: E1029 23:13:24.379102 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwn99" podUID="54576e13-0668-4255-952b-46b2bf5a279c" Oct 29 23:13:26.273913 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.273883 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeReady" Oct 29 23:13:26.274289 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.274030 2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 29 23:13:26.330603 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.330574 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5bzwq"] Oct 29 23:13:26.332413 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.332396 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:26.333979 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.333947 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7g2cz"] Oct 29 23:13:26.335917 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.335898 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7g2cz" Oct 29 23:13:26.351391 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.351364 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mmg8f"] Oct 29 23:13:26.353216 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.353201 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.353770 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.353736 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 23:13:26.353866 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.353784 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 23:13:26.353866 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.353854 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tldxj\"" Oct 29 23:13:26.356105 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.356085 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 23:13:26.356345 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.356328 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 23:13:26.364670 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.364644 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gj5kk\"" Oct 29 23:13:26.364770 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.364720 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 23:13:26.366060 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.366038 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5bzwq"] Oct 29 23:13:26.366939 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.366922 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7g2cz"] Oct 29 23:13:26.376340 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.376302 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mmg8f"] Oct 29 23:13:26.378816 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.378795 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:26.378914 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.378814 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:26.378970 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.378941 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:26.454162 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.454123 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 23:13:26.454327 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.454181 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 23:13:26.454327 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.454242 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 23:13:26.454327 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.454300 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 23:13:26.454327 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.454300 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 23:13:26.454327 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.454305 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 23:13:26.454536 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.454393 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 23:13:26.454536 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.454399 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sx9m6\"" Oct 29 23:13:26.454536 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.454419 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2vcdm\"" Oct 29 23:13:26.454536 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.454430 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-89z4k\"" Oct 29 23:13:26.454536 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.454520 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-phkn8\"" Oct 29 23:13:26.454685 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.454544 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 23:13:26.462909 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.462885 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a00edc7-9624-4236-bb7c-9ac71e911416-data-volume\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.463095 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.463070 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/916cb471-df04-4560-aaea-7e9d27f41897-config-volume\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:26.463193 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.463099 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qspl\" (UniqueName: \"kubernetes.io/projected/85acd560-02f0-4df7-b672-f0c099281e7a-kube-api-access-4qspl\") pod \"ingress-canary-7g2cz\" (UID: \"85acd560-02f0-4df7-b672-f0c099281e7a\") " pod="openshift-ingress-canary/ingress-canary-7g2cz" Oct 29 23:13:26.463193 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.463133 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916cb471-df04-4560-aaea-7e9d27f41897-tmp-dir\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:26.463193 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.463157 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85acd560-02f0-4df7-b672-f0c099281e7a-cert\") pod \"ingress-canary-7g2cz\" (UID: \"85acd560-02f0-4df7-b672-f0c099281e7a\") " pod="openshift-ingress-canary/ingress-canary-7g2cz" Oct 29 23:13:26.463344 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.463199 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k4q5\" (UniqueName: \"kubernetes.io/projected/7a00edc7-9624-4236-bb7c-9ac71e911416-kube-api-access-9k4q5\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.463344 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.463245 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a00edc7-9624-4236-bb7c-9ac71e911416-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.463344 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.463316 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a00edc7-9624-4236-bb7c-9ac71e911416-crio-socket\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.463447 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.463362 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-858x5\" (UniqueName: \"kubernetes.io/projected/916cb471-df04-4560-aaea-7e9d27f41897-kube-api-access-858x5\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:26.463447 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.463377 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a00edc7-9624-4236-bb7c-9ac71e911416-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.463447 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.463404 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/916cb471-df04-4560-aaea-7e9d27f41897-metrics-tls\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:26.564788 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.564705 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/916cb471-df04-4560-aaea-7e9d27f41897-config-volume\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:26.564788 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.564741 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qspl\" (UniqueName: \"kubernetes.io/projected/85acd560-02f0-4df7-b672-f0c099281e7a-kube-api-access-4qspl\") pod \"ingress-canary-7g2cz\" (UID: \"85acd560-02f0-4df7-b672-f0c099281e7a\") " pod="openshift-ingress-canary/ingress-canary-7g2cz" Oct 29 23:13:26.564788 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.564765 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916cb471-df04-4560-aaea-7e9d27f41897-tmp-dir\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:26.565052 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.564967 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85acd560-02f0-4df7-b672-f0c099281e7a-cert\") pod \"ingress-canary-7g2cz\" (UID: \"85acd560-02f0-4df7-b672-f0c099281e7a\") " pod="openshift-ingress-canary/ingress-canary-7g2cz" Oct 29 23:13:26.565052 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.565001 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k4q5\" (UniqueName: \"kubernetes.io/projected/7a00edc7-9624-4236-bb7c-9ac71e911416-kube-api-access-9k4q5\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.565052 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.565031 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a00edc7-9624-4236-bb7c-9ac71e911416-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.565215 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.565082 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a00edc7-9624-4236-bb7c-9ac71e911416-crio-socket\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.565215 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.565101 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916cb471-df04-4560-aaea-7e9d27f41897-tmp-dir\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:26.565335 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.565230 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-858x5\" (UniqueName: \"kubernetes.io/projected/916cb471-df04-4560-aaea-7e9d27f41897-kube-api-access-858x5\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:26.565335 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.565267 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a00edc7-9624-4236-bb7c-9ac71e911416-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.565335 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.565242 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a00edc7-9624-4236-bb7c-9ac71e911416-crio-socket\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.565335 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.565320 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/916cb471-df04-4560-aaea-7e9d27f41897-metrics-tls\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:26.565562 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.565342 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a00edc7-9624-4236-bb7c-9ac71e911416-data-volume\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.565562 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.565482 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/916cb471-df04-4560-aaea-7e9d27f41897-config-volume\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:26.565757 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.565733 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a00edc7-9624-4236-bb7c-9ac71e911416-data-volume\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.565920 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.565903 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a00edc7-9624-4236-bb7c-9ac71e911416-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.568758 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.568728 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/916cb471-df04-4560-aaea-7e9d27f41897-metrics-tls\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:26.568909 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.568892 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a00edc7-9624-4236-bb7c-9ac71e911416-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.569091 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.569071 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85acd560-02f0-4df7-b672-f0c099281e7a-cert\") pod \"ingress-canary-7g2cz\" (UID: \"85acd560-02f0-4df7-b672-f0c099281e7a\") " pod="openshift-ingress-canary/ingress-canary-7g2cz" Oct 29 23:13:26.585895 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.585866 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k4q5\" (UniqueName: \"kubernetes.io/projected/7a00edc7-9624-4236-bb7c-9ac71e911416-kube-api-access-9k4q5\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.586009 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.585898 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-858x5\" (UniqueName: \"kubernetes.io/projected/916cb471-df04-4560-aaea-7e9d27f41897-kube-api-access-858x5\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:26.586009 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.585874 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qspl\" (UniqueName: \"kubernetes.io/projected/85acd560-02f0-4df7-b672-f0c099281e7a-kube-api-access-4qspl\") pod \"ingress-canary-7g2cz\" (UID: \"85acd560-02f0-4df7-b672-f0c099281e7a\") " pod="openshift-ingress-canary/ingress-canary-7g2cz" Oct 29 23:13:26.645899 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.645861 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:26.649209 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.649193 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7g2cz" Oct 29 23:13:26.666923 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.666893 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:26.984237 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.984200 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7g2cz"] Oct 29 23:13:26.984873 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:26.984848 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5bzwq"] Oct 29 23:13:26.988444 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:13:26.988418 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916cb471_df04_4560_aaea_7e9d27f41897.slice/crio-889f0098da894fab68152e20bf4a1979512d78ab550794b83e701dba7de9ffc8 WatchSource:0}: Error finding container 889f0098da894fab68152e20bf4a1979512d78ab550794b83e701dba7de9ffc8: Status 404 returned error can't find the container with id 889f0098da894fab68152e20bf4a1979512d78ab550794b83e701dba7de9ffc8 Oct 29 23:13:26.988678 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:13:26.988654 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85acd560_02f0_4df7_b672_f0c099281e7a.slice/crio-32883262f69fb005c5fba90b36f47abc0e94649e057d04cc200e2065222ee7db WatchSource:0}: Error finding container 32883262f69fb005c5fba90b36f47abc0e94649e057d04cc200e2065222ee7db: Status 404 returned error can't find the container with id 32883262f69fb005c5fba90b36f47abc0e94649e057d04cc200e2065222ee7db Oct 29 23:13:27.000268 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:27.000244 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mmg8f"] Oct 29 23:13:27.002765 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:13:27.002731 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a00edc7_9624_4236_bb7c_9ac71e911416.slice/crio-69db1b82499ae56bbb1741cadf9f71338ce3ed100baf3ca2b0499152ccd7f52c WatchSource:0}: Error finding container 69db1b82499ae56bbb1741cadf9f71338ce3ed100baf3ca2b0499152ccd7f52c: Status 404 returned error can't find the container with id 69db1b82499ae56bbb1741cadf9f71338ce3ed100baf3ca2b0499152ccd7f52c Oct 29 23:13:27.697974 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:27.697938 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mmg8f" event={"ID":"7a00edc7-9624-4236-bb7c-9ac71e911416","Type":"ContainerStarted","Data":"69c3a1c9ff6e7249a8bfb6019eb03e4422347e06638f16cf4f81297f39ed66e7"} Oct 29 23:13:27.698390 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:27.698011 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mmg8f" event={"ID":"7a00edc7-9624-4236-bb7c-9ac71e911416","Type":"ContainerStarted","Data":"56dcd70d0b69e5e837b71c9665d4a9baf5da2a0f4fc92c41b81e445e02fd4e02"} Oct 29 23:13:27.698390 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:27.698040 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mmg8f" event={"ID":"7a00edc7-9624-4236-bb7c-9ac71e911416","Type":"ContainerStarted","Data":"69db1b82499ae56bbb1741cadf9f71338ce3ed100baf3ca2b0499152ccd7f52c"} Oct 29 23:13:27.699172 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:27.699147 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7g2cz" event={"ID":"85acd560-02f0-4df7-b672-f0c099281e7a","Type":"ContainerStarted","Data":"32883262f69fb005c5fba90b36f47abc0e94649e057d04cc200e2065222ee7db"} Oct 29 23:13:27.700479 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:27.700442 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5bzwq" event={"ID":"916cb471-df04-4560-aaea-7e9d27f41897","Type":"ContainerStarted","Data":"889f0098da894fab68152e20bf4a1979512d78ab550794b83e701dba7de9ffc8"} Oct 29 23:13:29.707440 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:29.707404 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5bzwq" event={"ID":"916cb471-df04-4560-aaea-7e9d27f41897","Type":"ContainerStarted","Data":"ff3989261241d9ffbe02d5d847e9532c303b3e4c211b28f5a4bbf5b819985782"} Oct 29 23:13:29.707871 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:29.707505 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:29.707871 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:29.707524 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5bzwq" event={"ID":"916cb471-df04-4560-aaea-7e9d27f41897","Type":"ContainerStarted","Data":"9effffc3d8ae2454c700ebb5bbc4ee5c99d1026a42ac3a759d77861e980318b5"} Oct 29 23:13:29.709692 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:29.709664 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mmg8f" event={"ID":"7a00edc7-9624-4236-bb7c-9ac71e911416","Type":"ContainerStarted","Data":"6d08e1a9d00a558ec800958db9b37db88b3eabaaaa76ec68bef47ae45ed1b5ba"} Oct 29 23:13:29.711075 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:29.711053 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7g2cz" event={"ID":"85acd560-02f0-4df7-b672-f0c099281e7a","Type":"ContainerStarted","Data":"f3a3ac660fed52f35343784e7e52fed59489caadb8fa692858ac017d50c9f9c3"} Oct 29 23:13:29.753993 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:29.753942 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-5bzwq" podStartSLOduration=2.026981082 podStartE2EDuration="3.753929072s" totalImagesPullingTime="1.72694799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:29.75314298 +0000 UTC m=+51.824671517" watchObservedRunningTime="2026-10-29 23:13:29.753929072 +0000 UTC m=+51.825457608" Oct 29 23:13:29.801031 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:29.800983 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mmg8f" podStartSLOduration=1.8387231819999998 podStartE2EDuration="3.800967412s" totalImagesPullingTime="1.96224423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:13:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:29.799894963 +0000 UTC m=+51.871423518" watchObservedRunningTime="2026-10-29 23:13:29.800967412 +0000 UTC m=+51.872495948" Oct 29 23:13:29.843275 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:29.843185 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7g2cz" podStartSLOduration=1.239970968 podStartE2EDuration="3.843173638s" totalImagesPullingTime="2.60320267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:29.842292036 +0000 UTC m=+51.913820591" watchObservedRunningTime="2026-10-29 23:13:29.843173638 +0000 UTC m=+51.914702203" Oct 29 23:13:31.360172 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:31.360139 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5bzwq_916cb471-df04-4560-aaea-7e9d27f41897/dns/0.log" Oct 29 23:13:31.560403 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:31.560373 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5bzwq_916cb471-df04-4560-aaea-7e9d27f41897/kube-rbac-proxy/0.log" Oct 29 23:13:32.760393 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:32.760359 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f456x_76e09503-45e8-4e94-9a3b-a5d291a3990f/dns-node-resolver/0.log" Oct 29 23:13:33.415149 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.415118 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-j467d"] Oct 29 23:13:33.417156 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.417139 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:33.418369 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.418342 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e3f3838-8917-45a1-a496-652392caa72c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:33.418548 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.418385 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e3f3838-8917-45a1-a496-652392caa72c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:33.418548 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.418497 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3f3838-8917-45a1-a496-652392caa72c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:33.418548 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.418529 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6flh\" (UniqueName: \"kubernetes.io/projected/7e3f3838-8917-45a1-a496-652392caa72c-kube-api-access-v6flh\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:33.453156 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.453127 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 29 23:13:33.453342 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.453201 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 23:13:33.453342 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.453305 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 23:13:33.453342 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.453305 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rkfgr\"" Oct 29 23:13:33.453342 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.453332 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 29 23:13:33.453527 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.453352 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 29 23:13:33.454034 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.454016 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-j467d"] Oct 29 23:13:33.469669 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.469648 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th"] Oct 29 23:13:33.471596 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.471578 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8qc8d"] Oct 29 23:13:33.471719 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.471702 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.473341 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.473325 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.503285 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.503261 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th"] Oct 29 23:13:33.516960 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.516929 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 29 23:13:33.517082 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.516973 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 29 23:13:33.519675 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.519657 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxt65\" (UniqueName: \"kubernetes.io/projected/8cccc782-d044-4032-ba8b-8f4d4cdab41b-kube-api-access-cxt65\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.519739 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.519699 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np8k4\" (UniqueName: \"kubernetes.io/projected/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-api-access-np8k4\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.519739 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.519725 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e3f3838-8917-45a1-a496-652392caa72c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:33.519844 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.519784 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.519899 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.519883 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.519957 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.519937 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3f3838-8917-45a1-a496-652392caa72c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:33.520016 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.519992 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-textfile\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.520067 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.520016 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.520067 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.520039 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-sys\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.520167 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.520065 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/73623961-3ffb-4492-a2f4-74a5c4b9551e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.520167 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.520112 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e3f3838-8917-45a1-a496-652392caa72c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:33.520167 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.520149 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.520312 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.520278 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-root\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.520349 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.520309 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.520389 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.520373 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-tls\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.520438 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.520408 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-wtmp\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.520526 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.520507 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6flh\" (UniqueName: \"kubernetes.io/projected/7e3f3838-8917-45a1-a496-652392caa72c-kube-api-access-v6flh\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:33.520568 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.520551 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cccc782-d044-4032-ba8b-8f4d4cdab41b-metrics-client-ca\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.520602 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.520580 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73623961-3ffb-4492-a2f4-74a5c4b9551e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.520762 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.520747 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e3f3838-8917-45a1-a496-652392caa72c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:33.522218 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.522188 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e3f3838-8917-45a1-a496-652392caa72c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:33.522301 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.522273 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3f3838-8917-45a1-a496-652392caa72c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:33.525693 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.525665 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 29 23:13:33.525693 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.525676 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 29 23:13:33.525846 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.525747 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 29 23:13:33.525846 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.525814 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vwjx7\"" Oct 29 23:13:33.525951 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.525874 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 29 23:13:33.526215 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.526199 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2vtzc\"" Oct 29 23:13:33.544947 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.544923 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6flh\" (UniqueName: \"kubernetes.io/projected/7e3f3838-8917-45a1-a496-652392caa72c-kube-api-access-v6flh\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:33.621857 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.621819 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-textfile\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.621857 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.621855 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.622057 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.621874 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-sys\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.622091 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622064 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-sys\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.622123 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622092 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/73623961-3ffb-4492-a2f4-74a5c4b9551e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.622123 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622116 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.622181 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622134 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-root\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.622181 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622151 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.622181 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622172 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-tls\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.622297 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622200 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-wtmp\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.622297 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622211 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-root\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.622297 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622231 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cccc782-d044-4032-ba8b-8f4d4cdab41b-metrics-client-ca\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.622297 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622169 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-textfile\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.622297 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622255 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73623961-3ffb-4492-a2f4-74a5c4b9551e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.622297 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622285 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxt65\" (UniqueName: \"kubernetes.io/projected/8cccc782-d044-4032-ba8b-8f4d4cdab41b-kube-api-access-cxt65\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.622659 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622333 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np8k4\" (UniqueName: \"kubernetes.io/projected/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-api-access-np8k4\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.622659 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622361 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.622659 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622404 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.622659 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622430 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-wtmp\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.622868 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.622679 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/73623961-3ffb-4492-a2f4-74a5c4b9551e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.623054 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.623034 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cccc782-d044-4032-ba8b-8f4d4cdab41b-metrics-client-ca\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.623187 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.623159 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.623349 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.623322 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73623961-3ffb-4492-a2f4-74a5c4b9551e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.623651 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.623628 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.624539 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.624510 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.624797 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.624773 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.624889 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.624866 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-tls\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.624927 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.624895 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.643934 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.643911 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np8k4\" (UniqueName: \"kubernetes.io/projected/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-api-access-np8k4\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.644086 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.644071 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxt65\" (UniqueName: \"kubernetes.io/projected/8cccc782-d044-4032-ba8b-8f4d4cdab41b-kube-api-access-cxt65\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:33.728943 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.728915 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:33.784792 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.784768 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:33.789926 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:33.789901 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:34.046298 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.046276 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-j467d"] Oct 29 23:13:34.048032 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:13:34.047999 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3f3838_8917_45a1_a496_652392caa72c.slice/crio-6b338b2473e9686cee8d57ff28c46cef29f826b8c5b63dfee498236169935e16 WatchSource:0}: Error finding container 6b338b2473e9686cee8d57ff28c46cef29f826b8c5b63dfee498236169935e16: Status 404 returned error can't find the container with id 6b338b2473e9686cee8d57ff28c46cef29f826b8c5b63dfee498236169935e16 Oct 29 23:13:34.094579 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.094557 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th"] Oct 29 23:13:34.097689 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:13:34.097664 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73623961_3ffb_4492_a2f4_74a5c4b9551e.slice/crio-7329f3e20c8f7d066f85974cb05a3ae14853b884989d9c551ac11aa1fb7d232c WatchSource:0}: Error finding container 7329f3e20c8f7d066f85974cb05a3ae14853b884989d9c551ac11aa1fb7d232c: Status 404 returned error can't find the container with id 7329f3e20c8f7d066f85974cb05a3ae14853b884989d9c551ac11aa1fb7d232c Oct 29 23:13:34.159514 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.159473 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tr7bl_6170bf16-e81d-40fd-a4a0-449213b94d19/node-ca/0.log" Oct 29 23:13:34.561782 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.561753 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7g2cz_85acd560-02f0-4df7-b672-f0c099281e7a/serve-healthcheck-canary/0.log" Oct 29 23:13:34.725727 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.725648 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qc8d" event={"ID":"8cccc782-d044-4032-ba8b-8f4d4cdab41b","Type":"ContainerStarted","Data":"594c86e5a3b9bd1d16f3ecd79f6e62d63636c7e4b7add4c127d1758db64d2f45"} Oct 29 23:13:34.728853 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.728819 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" event={"ID":"7e3f3838-8917-45a1-a496-652392caa72c","Type":"ContainerStarted","Data":"2e92dc5857a339236e4de60d21913cf63c662368e530d1c1d2b26c51a8ce4275"} Oct 29 23:13:34.729006 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.728886 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" event={"ID":"7e3f3838-8917-45a1-a496-652392caa72c","Type":"ContainerStarted","Data":"259777e69c992e50583f0ce1d772d15cf7c98c83f3b1a5cd80dd5f327754abe3"} Oct 29 23:13:34.729006 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.728915 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" event={"ID":"7e3f3838-8917-45a1-a496-652392caa72c","Type":"ContainerStarted","Data":"6b338b2473e9686cee8d57ff28c46cef29f826b8c5b63dfee498236169935e16"} Oct 29 23:13:34.730704 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.730677 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" event={"ID":"73623961-3ffb-4492-a2f4-74a5c4b9551e","Type":"ContainerStarted","Data":"7329f3e20c8f7d066f85974cb05a3ae14853b884989d9c551ac11aa1fb7d232c"} Oct 29 23:13:34.863403 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.863378 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 23:13:34.865624 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.865603 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:34.886805 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.886780 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 23:13:34.896334 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.896311 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 29 23:13:34.896452 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.896405 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 29 23:13:34.896452 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.896311 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vvv78\"" Oct 29 23:13:34.896544 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.896452 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 29 23:13:34.899282 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.899262 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 29 23:13:34.899428 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.899410 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 29 23:13:34.899523 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.899484 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 29 23:13:34.899582 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.899574 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 29 23:13:34.899674 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.899659 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 29 23:13:34.900959 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.900939 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 29 23:13:34.934525 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.934488 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:34.934683 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.934575 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-config-out\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:34.934683 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.934607 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:34.934683 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.934651 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:34.934683 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.934677 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:34.934814 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.934707 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-web-config\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:34.934814 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.934744 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:34.934814 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.934780 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:34.934814 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.934807 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:34.934923 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.934820 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnbnn\" (UniqueName: \"kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-kube-api-access-tnbnn\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:34.934923 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.934843 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:34.934923 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.934860 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:34.934923 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:34.934886 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.035932 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.035889 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.036117 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.035952 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-config-out\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.036117 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.035973 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.036117 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.036007 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.036117 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.036032 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.036117 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.036064 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-web-config\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.036117 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.036084 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.036117 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.036118 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.036551 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.036149 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.036551 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.036167 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbnn\" (UniqueName: \"kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-kube-api-access-tnbnn\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.036551 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.036191 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.036551 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.036206 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.036551 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.036245 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.036920 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.036893 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.037007 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.036897 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.037902 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.037815 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.040802 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.040775 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.042132 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.042112 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-web-config\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.043123 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.043103 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.043206 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.043170 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.043511 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.043324 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.043599 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.043565 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.043719 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.043660 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.044856 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.044831 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-config-out\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.062408 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.062363 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.065636 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.065585 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnbnn\" (UniqueName: \"kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-kube-api-access-tnbnn\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.191124 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.191096 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:35.529632 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.529603 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 23:13:35.533238 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:13:35.533203 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e2d17bd_2440_419f_9403_73f10f421359.slice/crio-6f56629e6314704d696b28a21f2fbf244aebd8c6100286458fcd858c4d37d12c WatchSource:0}: Error finding container 6f56629e6314704d696b28a21f2fbf244aebd8c6100286458fcd858c4d37d12c: Status 404 returned error can't find the container with id 6f56629e6314704d696b28a21f2fbf244aebd8c6100286458fcd858c4d37d12c Oct 29 23:13:35.737016 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.736979 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" event={"ID":"7e3f3838-8917-45a1-a496-652392caa72c","Type":"ContainerStarted","Data":"9fb30cbe3820c9ce18b95d2da0d3eb9890138ce9b0ea09fbc39e96bea9cd69d5"} Oct 29 23:13:35.739188 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.739157 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" event={"ID":"73623961-3ffb-4492-a2f4-74a5c4b9551e","Type":"ContainerStarted","Data":"00bd812500c912a43d25b4adee09e840234f33994633be4380043e5c1e6c2d96"} Oct 29 23:13:35.739318 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.739208 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" event={"ID":"73623961-3ffb-4492-a2f4-74a5c4b9551e","Type":"ContainerStarted","Data":"5d1fd22cff3aea0131a41963e854b2b8de6ae695a3e96d35dea3fda0ab2aa73b"} Oct 29 23:13:35.739318 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.739229 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" event={"ID":"73623961-3ffb-4492-a2f4-74a5c4b9551e","Type":"ContainerStarted","Data":"4930d9fbed17f9d647f6c69420439af2a70c5fba5030e794b31c4e085c3747c9"} Oct 29 23:13:35.740390 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.740368 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e2d17bd-2440-419f-9403-73f10f421359","Type":"ContainerStarted","Data":"6f56629e6314704d696b28a21f2fbf244aebd8c6100286458fcd858c4d37d12c"} Oct 29 23:13:35.741948 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.741916 2571 generic.go:356] "Generic (PLEG): container finished" podID="8cccc782-d044-4032-ba8b-8f4d4cdab41b" containerID="5dd671a85d692728e2fe38cd21b44ad5d0d124939559070f52276a9139154811" exitCode=0 Oct 29 23:13:35.742032 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.741969 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qc8d" event={"ID":"8cccc782-d044-4032-ba8b-8f4d4cdab41b","Type":"ContainerDied","Data":"5dd671a85d692728e2fe38cd21b44ad5d0d124939559070f52276a9139154811"} Oct 29 23:13:35.785992 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.785940 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" podStartSLOduration=1.819773096 podStartE2EDuration="2.785923621s" totalImagesPullingTime="966.150525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:35.784051515 +0000 UTC m=+57.855580051" watchObservedRunningTime="2026-10-29 23:13:35.785923621 +0000 UTC m=+57.857452156" Oct 29 23:13:35.865774 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:35.865727 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" podStartSLOduration=1.699522703 podStartE2EDuration="2.865711631s" totalImagesPullingTime="1.166188928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:35.864524327 +0000 UTC m=+57.936052860" watchObservedRunningTime="2026-10-29 23:13:35.865711631 +0000 UTC m=+57.937240166" Oct 29 23:13:36.746891 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:36.746795 2571 generic.go:356] "Generic (PLEG): container finished" podID="6e2d17bd-2440-419f-9403-73f10f421359" containerID="eafbb4d1a4d3ec0c0d9881397c0f6acbc4799761dde36166a54b45df8fd53bf1" exitCode=0 Oct 29 23:13:36.746891 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:36.746883 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e2d17bd-2440-419f-9403-73f10f421359","Type":"ContainerDied","Data":"eafbb4d1a4d3ec0c0d9881397c0f6acbc4799761dde36166a54b45df8fd53bf1"} Oct 29 23:13:36.752974 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:36.752933 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qc8d" event={"ID":"8cccc782-d044-4032-ba8b-8f4d4cdab41b","Type":"ContainerStarted","Data":"292d030cf4f1c23faeb9523104a6b13804f4cddab38b32bc24f22ea8fbbd7635"} Oct 29 23:13:36.753104 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:36.753064 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qc8d" event={"ID":"8cccc782-d044-4032-ba8b-8f4d4cdab41b","Type":"ContainerStarted","Data":"6099c0a64b63ee90a5d3bc362ee854436b2c08dfaf6fb1d695ad53846b9fb50e"} Oct 29 23:13:36.845936 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:36.845880 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8qc8d" podStartSLOduration=2.769682338 podStartE2EDuration="3.845865396s" totalImagesPullingTime="990.978587ms" totalInitContainerRuntime="85.204471ms" isStatefulPod=true podCreationTimestamp="2026-10-29 23:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:36.844429934 +0000 UTC m=+58.915958492" watchObservedRunningTime="2026-10-29 23:13:36.845865396 +0000 UTC m=+58.917393926" Oct 29 23:13:38.764129 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:38.764090 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e2d17bd-2440-419f-9403-73f10f421359","Type":"ContainerStarted","Data":"e01f48c65de1075865455a098a5d1241260e5af5c6f4df20a79ffd89e9be4b5e"} Oct 29 23:13:38.764555 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:38.764161 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e2d17bd-2440-419f-9403-73f10f421359","Type":"ContainerStarted","Data":"372657f190070a2029a6036dd7ad3e45a76fddc8ed6e9fdad04fecacd5b50cda"} Oct 29 23:13:38.764555 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:38.764194 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e2d17bd-2440-419f-9403-73f10f421359","Type":"ContainerStarted","Data":"147c61fdc70a55c617acd2b28d32e2e3b32e694f38e2f2fd9ea28692cffce801"} Oct 29 23:13:38.764555 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:38.764220 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e2d17bd-2440-419f-9403-73f10f421359","Type":"ContainerStarted","Data":"aea55638adf89557b3d2f1a784c52c2cda19c440a6f930fa5d3d7bb6d8cfca8a"} Oct 29 23:13:38.764555 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:38.764242 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e2d17bd-2440-419f-9403-73f10f421359","Type":"ContainerStarted","Data":"59250c65c1ead7102d5de10cea540bcec94919334a6eccb71632a3760316cdcf"} Oct 29 23:13:39.612153 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.612123 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cc6cb8466-tphkb"] Oct 29 23:13:39.615364 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.615348 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.632445 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.632418 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 29 23:13:39.633052 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.633037 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 29 23:13:39.633159 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.633141 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 29 23:13:39.646567 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.646543 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 29 23:13:39.675516 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.675490 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 29 23:13:39.675678 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.675522 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 29 23:13:39.675678 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.675540 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 29 23:13:39.675678 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.675537 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gnrh6\"" Oct 29 23:13:39.675678 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.675594 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 29 23:13:39.676582 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.676564 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cc6cb8466-tphkb"] Oct 29 23:13:39.679016 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.679000 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.679082 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.679034 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.679082 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.679055 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.679082 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.679073 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/defafe8e-3565-464e-b2be-522d526248a4-tmp\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.679192 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.679114 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnbnk\" (UniqueName: \"kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.679192 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.679170 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.679274 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.679210 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.679274 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.679231 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.715113 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.715088 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:39.771208 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.771177 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e2d17bd-2440-419f-9403-73f10f421359","Type":"ContainerStarted","Data":"1e5212ad899b6158f44ac73409016545410d98f0b5c9504631292948e52e86d9"} Oct 29 23:13:39.779848 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.779821 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.779848 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.779852 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.780046 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.779922 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.780046 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.779976 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.780046 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.780020 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.780046 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.780045 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/defafe8e-3565-464e-b2be-522d526248a4-tmp\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.780238 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.780067 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnbnk\" (UniqueName: \"kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.780238 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.780095 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.780769 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.780742 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.780877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.780861 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.781006 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.780988 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/defafe8e-3565-464e-b2be-522d526248a4-tmp\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.781246 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.781225 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.781298 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.781275 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.782576 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.782552 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.782686 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.782596 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.803328 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.801684 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnbnk\" (UniqueName: \"kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:39.822551 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.822501 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.051272492 podStartE2EDuration="5.82244953s" totalImagesPullingTime="2.759986388s" totalInitContainerRuntime="11.19065ms" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:39.821522633 +0000 UTC m=+61.893051192" watchObservedRunningTime="2026-10-29 23:13:39.82244953 +0000 UTC m=+61.893978065" Oct 29 23:13:39.926772 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:39.926674 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:40.229172 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.229151 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cc6cb8466-tphkb"] Oct 29 23:13:40.231300 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:13:40.231272 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefafe8e_3565_464e_b2be_522d526248a4.slice/crio-66c61446fb25e53f3d0ae67b70a4dbd35e26e55bdc8fba9858feb4cd843c4d7b WatchSource:0}: Error finding container 66c61446fb25e53f3d0ae67b70a4dbd35e26e55bdc8fba9858feb4cd843c4d7b: Status 404 returned error can't find the container with id 66c61446fb25e53f3d0ae67b70a4dbd35e26e55bdc8fba9858feb4cd843c4d7b Oct 29 23:13:40.414868 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.413979 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 23:13:40.419755 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.419731 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.442828 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.442799 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 29 23:13:40.442939 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.442843 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 29 23:13:40.443394 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.443375 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 29 23:13:40.443394 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.443379 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 29 23:13:40.443550 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.443438 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 29 23:13:40.443550 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.443506 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 29 23:13:40.443550 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.443508 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 29 23:13:40.443550 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.443506 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 29 23:13:40.443550 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.443548 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ejptm05c9424c\"" Oct 29 23:13:40.443715 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.443600 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 29 23:13:40.443715 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.443701 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 29 23:13:40.443845 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.443811 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 29 23:13:40.457130 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.457110 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-59hcs\"" Oct 29 23:13:40.457303 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.457290 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 29 23:13:40.457303 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.457296 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 29 23:13:40.471799 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.471776 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 29 23:13:40.471888 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.471825 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 29 23:13:40.473016 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.472995 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 23:13:40.488802 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.488746 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.488802 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.488783 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.488936 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.488888 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.488936 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.488913 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.489029 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.488939 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.489029 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.488957 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.489115 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.489023 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.489161 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.489153 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.489193 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.489172 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.489193 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.489185 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.489282 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.489210 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.489282 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.489227 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.489282 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.489256 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.489420 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.489336 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.489420 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.489374 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cxhf\" (UniqueName: \"kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-kube-api-access-6cxhf\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.489539 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.489443 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.489539 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.489494 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.489600 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.489540 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.489600 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.489565 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.489600 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.489583 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590273 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590240 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590273 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590274 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590518 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590294 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590518 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590311 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590518 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590335 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590518 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590359 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590518 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590393 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590518 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590417 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590518 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590445 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590873 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590533 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590873 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590556 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590873 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590577 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590873 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590617 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590873 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590645 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590873 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590670 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590873 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590707 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590873 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590732 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxhf\" (UniqueName: \"kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-kube-api-access-6cxhf\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590873 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590780 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590873 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590808 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.590873 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.590848 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.591392 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.591172 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.592065 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.592034 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.593331 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.593306 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.593533 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.593514 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.593990 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.593965 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.594149 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.594124 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.595024 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.594590 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.595024 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.594808 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.595024 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.594877 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.595024 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.594978 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.595262 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.595116 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.595978 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.595958 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.596092 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.596072 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.596259 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.596236 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.596344 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.596249 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.596674 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.596639 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.596674 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.596668 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.597300 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.597270 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.597814 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.597795 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.613306 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.613278 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxhf\" (UniqueName: \"kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-kube-api-access-6cxhf\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.744541 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.744447 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:40.776506 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:40.776450 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc6cb8466-tphkb" event={"ID":"defafe8e-3565-464e-b2be-522d526248a4","Type":"ContainerStarted","Data":"66c61446fb25e53f3d0ae67b70a4dbd35e26e55bdc8fba9858feb4cd843c4d7b"} Oct 29 23:13:41.091689 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:41.091656 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 23:13:41.093893 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:13:41.093842 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec1c1e5_dfa8_4b8b_bb97_e9bd48427ffc.slice/crio-f47734bfb9a7143130d8cb58a4a3e3999c5b8a5ed3b82f5ae191b1c8babc1178 WatchSource:0}: Error finding container f47734bfb9a7143130d8cb58a4a3e3999c5b8a5ed3b82f5ae191b1c8babc1178: Status 404 returned error can't find the container with id f47734bfb9a7143130d8cb58a4a3e3999c5b8a5ed3b82f5ae191b1c8babc1178 Oct 29 23:13:41.781964 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:41.781921 2571 generic.go:356] "Generic (PLEG): container finished" podID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerID="58271d5668a0a40f2a7a29fde78624521d20d0b178611b7075a8e004751862dc" exitCode=0 Oct 29 23:13:41.782435 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:41.782020 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc","Type":"ContainerDied","Data":"58271d5668a0a40f2a7a29fde78624521d20d0b178611b7075a8e004751862dc"} Oct 29 23:13:41.782435 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:41.782135 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc","Type":"ContainerStarted","Data":"f47734bfb9a7143130d8cb58a4a3e3999c5b8a5ed3b82f5ae191b1c8babc1178"} Oct 29 23:13:43.790896 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:43.790858 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc6cb8466-tphkb" event={"ID":"defafe8e-3565-464e-b2be-522d526248a4","Type":"ContainerStarted","Data":"be2eaccc4c8d7c97006308ffd516c6049d44fdde0f5d35894ba6eb42dec87ee8"} Oct 29 23:13:43.838430 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:43.838362 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5cc6cb8466-tphkb" podStartSLOduration=2.204859216 podStartE2EDuration="4.838341385s" totalImagesPullingTime="2.633482169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:43.836971723 +0000 UTC m=+65.908500259" watchObservedRunningTime="2026-10-29 23:13:43.838341385 +0000 UTC m=+65.909869922" Oct 29 23:13:44.226378 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.226340 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:44.226575 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.226398 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:44.242713 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.242681 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 23:13:44.242856 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.242798 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 23:13:44.249381 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.249360 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:44.249509 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.249425 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:44.327738 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.327692 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56n47\" (UniqueName: \"kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47\") pod \"network-check-target-69jbt\" (UID: \"0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc\") " pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:44.343692 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.343663 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 23:13:44.364148 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.364120 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 23:13:44.371938 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.371918 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56n47\" (UniqueName: \"kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47\") pod \"network-check-target-69jbt\" (UID: \"0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc\") " pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:44.409023 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.409001 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2vcdm\"" Oct 29 23:13:44.414173 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.414149 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:44.414920 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.414899 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-phkn8\"" Oct 29 23:13:44.419594 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.419573 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:44.420146 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.420128 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-89z4k\"" Oct 29 23:13:44.425438 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.425417 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:44.771852 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.771827 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xwn99"] Oct 29 23:13:44.773435 ip-10-0-133-100 kubenswrapper[2571]: W1029 23:13:44.773406 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54576e13_0668_4255_952b_46b2bf5a279c.slice/crio-ccc0103ba9931583242315b5593b1382f02a9686b020724476322573c08170fa WatchSource:0}: Error finding container ccc0103ba9931583242315b5593b1382f02a9686b020724476322573c08170fa: Status 404 returned error can't find the container with id ccc0103ba9931583242315b5593b1382f02a9686b020724476322573c08170fa Oct 29 23:13:44.780575 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.780552 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-69jbt"] Oct 29 23:13:44.781947 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.781918 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xcszc"] Oct 29 23:13:44.800216 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.800186 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc","Type":"ContainerStarted","Data":"dc89d501d1ea72f9863d2f40ba87257fe8fdb40ce4985025e9635b09edf3e273"} Oct 29 23:13:44.800636 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.800299 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc","Type":"ContainerStarted","Data":"7feb7ca52668f5d3582ea64bbb6b6d20d0acd0b301050b221fbe584abee58722"} Oct 29 23:13:44.801615 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.801582 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xcszc" event={"ID":"254e83a6-ab81-4b6b-b5ce-0538ccaede49","Type":"ContainerStarted","Data":"adb4703c15ab556bbd5cd4bd0dbf1b92872238ccfb62730d4009dd77804760a6"} Oct 29 23:13:44.802867 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.802846 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-69jbt" event={"ID":"0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc","Type":"ContainerStarted","Data":"b88d86003c8c53f05e764fb2a4528ce7149004260b8b362e64002cafc88871d4"} Oct 29 23:13:44.804009 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:44.803990 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwn99" event={"ID":"54576e13-0668-4255-952b-46b2bf5a279c","Type":"ContainerStarted","Data":"ccc0103ba9931583242315b5593b1382f02a9686b020724476322573c08170fa"} Oct 29 23:13:46.821125 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:46.821086 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwn99" event={"ID":"54576e13-0668-4255-952b-46b2bf5a279c","Type":"ContainerStarted","Data":"3661f125015f1106e159c48a96ec00dc666130be411788ae78ee7df2c51c0cd0"} Oct 29 23:13:46.821796 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:46.821154 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwn99" event={"ID":"54576e13-0668-4255-952b-46b2bf5a279c","Type":"ContainerStarted","Data":"6c4606c544aa2b879a0e44411298d43066e0263cdae9eaecda4fd100ef64c049"} Oct 29 23:13:46.864363 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:46.863118 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xwn99" podStartSLOduration=67.851235155 podStartE2EDuration="1m8.863099344s" totalImagesPullingTime="1.011864189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:46.861808405 +0000 UTC m=+68.933336942" watchObservedRunningTime="2026-10-29 23:13:46.863099344 +0000 UTC m=+68.934627881" Oct 29 23:13:47.022398 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:47.022362 2571 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cc6cb8466-tphkb"] Oct 29 23:13:47.830513 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:47.830469 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc","Type":"ContainerStarted","Data":"e1cab712bd585284b0e4ef4654f239e190a4a7703c50d4c8d9ec9b19dd6d60cc"} Oct 29 23:13:47.830964 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:47.830567 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc","Type":"ContainerStarted","Data":"488b57da1b9c5521c3e976682973b7f53163694e80170705a6f27c5529ca3776"} Oct 29 23:13:47.830964 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:47.830609 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc","Type":"ContainerStarted","Data":"f26ac97deff254d52616bbafe03846b0145321f1201a42d6eeb5dc12163bde80"} Oct 29 23:13:47.830964 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:47.830651 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc","Type":"ContainerStarted","Data":"c39ce6033548ccafb4f26caaa178b83e737b80ba5968f1879906884bc93a8960"} Oct 29 23:13:47.917069 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:47.917013 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.662828421 podStartE2EDuration="7.91699559s" totalImagesPullingTime="5.232889995s" totalInitContainerRuntime="21.277174ms" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:47.901282531 +0000 UTC m=+69.972811068" watchObservedRunningTime="2026-10-29 23:13:47.91699559 +0000 UTC m=+69.988524126" Oct 29 23:13:48.837363 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:48.837268 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-69jbt" event={"ID":"0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc","Type":"ContainerStarted","Data":"eaa0bb6cb63d9ae08802c788ffa9c2ab42126a31d1e09b3a04890a107c0eeeb8"} Oct 29 23:13:48.837363 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:48.837331 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:48.880715 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:48.880634 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-69jbt" podStartSLOduration=67.205536269 podStartE2EDuration="1m10.880614755s" totalImagesPullingTime="3.675078486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:48.879301839 +0000 UTC m=+70.950830375" watchObservedRunningTime="2026-10-29 23:13:48.880614755 +0000 UTC m=+70.952143292" Oct 29 23:13:49.927877 ip-10-0-133-100 kubenswrapper[2571]: I1029 23:13:49.927801 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:50.558262 ip-10-0-133-100 systemd[1]: Stopping Kubernetes Kubelet... Oct 29 23:13:50.581009 ip-10-0-133-100 systemd[1]: kubelet.service: Deactivated successfully. Oct 29 23:13:50.581247 ip-10-0-133-100 systemd[1]: Stopped Kubernetes Kubelet. Oct 29 23:13:50.581307 ip-10-0-133-100 systemd[1]: kubelet.service: Consumed 4.232s CPU time. Oct 29 23:13:50.597980 ip-10-0-133-100 systemd[1]: Starting Kubernetes Kubelet... Oct 29 23:13:50.667633 ip-10-0-133-100 kubenswrapper[6507]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 23:13:50.667633 ip-10-0-133-100 kubenswrapper[6507]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 23:13:50.667633 ip-10-0-133-100 kubenswrapper[6507]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 23:13:50.667633 ip-10-0-133-100 kubenswrapper[6507]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 23:13:50.669695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669607 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 23:13:50.669695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669695 6507 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 23:13:50.669695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669701 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669705 6507 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669709 6507 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669712 6507 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669715 6507 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669718 6507 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669720 6507 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669723 6507 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669726 6507 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669728 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669731 6507 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669734 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669737 6507 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669739 6507 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669742 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669745 6507 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669748 6507 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669750 6507 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669754 6507 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669757 6507 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 23:13:50.669807 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669759 6507 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669762 6507 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669765 6507 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669768 6507 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669771 6507 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669773 6507 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669783 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669786 6507 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669789 6507 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669792 6507 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669794 6507 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669797 6507 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669800 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669803 6507 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669805 6507 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669808 6507 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669813 6507 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669817 6507 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669820 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 23:13:50.670294 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669823 6507 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669826 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669829 6507 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669832 6507 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669834 6507 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669837 6507 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669839 6507 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669843 6507 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669846 6507 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669848 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669851 6507 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669854 6507 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669856 6507 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669859 6507 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669862 6507 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669865 6507 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669868 6507 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669870 6507 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669873 6507 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669875 6507 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 23:13:50.670753 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669878 6507 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669881 6507 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669883 6507 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669886 6507 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669888 6507 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669891 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669893 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669896 6507 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669898 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669901 6507 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669903 6507 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669905 6507 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669908 6507 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669910 6507 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669913 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669915 6507 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669918 6507 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669920 6507 feature_gate.go:386] unrecognized feature gate: Example Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669923 6507 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669926 6507 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 23:13:50.671228 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669929 6507 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669932 6507 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669934 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669937 6507 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669940 6507 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669943 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669946 6507 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669948 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669951 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669954 6507 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669956 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669958 6507 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669961 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669964 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669966 6507 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669969 6507 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669971 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669974 6507 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669977 6507 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 23:13:50.671736 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669979 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669982 6507 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669985 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669988 6507 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669990 6507 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.669993 6507 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670761 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670775 6507 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670779 6507 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670783 6507 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670785 6507 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670789 6507 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670792 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670795 6507 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670799 6507 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670802 6507 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670804 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670807 6507 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670810 6507 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 23:13:50.672187 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670812 6507 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670815 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670817 6507 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670820 6507 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670822 6507 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670825 6507 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670828 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670830 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670833 6507 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670836 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670839 6507 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670841 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670844 6507 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670846 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670849 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670852 6507 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670854 6507 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670858 6507 feature_gate.go:386] unrecognized feature gate: Example Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670860 6507 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 23:13:50.672695 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670863 6507 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670865 6507 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670868 6507 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670870 6507 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670873 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670876 6507 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670879 6507 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670881 6507 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670884 6507 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670887 6507 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670890 6507 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670892 6507 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670895 6507 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670897 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670900 6507 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670902 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670905 6507 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670907 6507 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670910 6507 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670913 6507 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 23:13:50.673139 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670915 6507 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670918 6507 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670921 6507 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670924 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670926 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670929 6507 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670932 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670935 6507 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670938 6507 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670940 6507 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670943 6507 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670945 6507 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670948 6507 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670951 6507 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670953 6507 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670956 6507 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670958 6507 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670961 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670964 6507 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 23:13:50.673637 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670967 6507 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670969 6507 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670972 6507 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670975 6507 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670978 6507 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670981 6507 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670984 6507 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670986 6507 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670989 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670991 6507 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670994 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670996 6507 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.670999 6507 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671001 6507 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671004 6507 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671006 6507 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671009 6507 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671013 6507 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671016 6507 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671019 6507 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 23:13:50.674117 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671023 6507 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671026 6507 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671031 6507 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671034 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671037 6507 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671040 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671043 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671049 6507 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671052 6507 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671054 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671057 6507 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671059 6507 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671062 6507 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671065 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671068 6507 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671621 6507 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671628 6507 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671632 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671634 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 23:13:50.674608 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671637 6507 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671640 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671643 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671645 6507 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671648 6507 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671650 6507 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671653 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671656 6507 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671659 6507 feature_gate.go:386] unrecognized feature gate: Example Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671661 6507 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671664 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671667 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671669 6507 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671672 6507 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671675 6507 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671677 6507 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671680 6507 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671682 6507 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671685 6507 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671687 6507 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 23:13:50.675065 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671692 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671695 6507 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671698 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671701 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671703 6507 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671707 6507 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671711 6507 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671714 6507 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671716 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671719 6507 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671722 6507 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671725 6507 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671728 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671730 6507 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671733 6507 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671735 6507 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671738 6507 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671740 6507 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671743 6507 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671746 6507 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 23:13:50.675559 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671749 6507 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671752 6507 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671754 6507 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671757 6507 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671760 6507 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671762 6507 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671765 6507 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671768 6507 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671771 6507 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671773 6507 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671776 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671779 6507 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671785 6507 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671787 6507 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671790 6507 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671793 6507 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671795 6507 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671798 6507 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671800 6507 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671803 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 23:13:50.676044 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671805 6507 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671807 6507 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671810 6507 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671813 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671816 6507 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671818 6507 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671821 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671823 6507 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671825 6507 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671828 6507 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671831 6507 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671835 6507 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671838 6507 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671841 6507 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671843 6507 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671846 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671848 6507 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671851 6507 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671854 6507 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 23:13:50.676537 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671857 6507 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671860 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671862 6507 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671865 6507 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671867 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671872 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671874 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671877 6507 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671880 6507 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671882 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671885 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671887 6507 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671890 6507 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671893 6507 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671895 6507 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671898 6507 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671901 6507 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671904 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671907 6507 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 23:13:50.676984 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671910 6507 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671912 6507 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671915 6507 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.671917 6507 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672014 6507 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672031 6507 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672041 6507 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672047 6507 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672052 6507 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672055 6507 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672059 6507 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672063 6507 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672066 6507 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672069 6507 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672073 6507 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672076 6507 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672079 6507 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672082 6507 flags.go:64] FLAG: --cgroup-root="" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672087 6507 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672090 6507 flags.go:64] FLAG: --client-ca-file="" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672093 6507 flags.go:64] FLAG: --cloud-provider="external" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672097 6507 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672101 6507 flags.go:64] FLAG: --cluster-domain="" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672104 6507 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 23:13:50.677436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672108 6507 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672111 6507 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672114 6507 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672118 6507 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672140 6507 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672144 6507 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672148 6507 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672152 6507 flags.go:64] FLAG: --contention-profiling="false" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672155 6507 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672158 6507 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672161 6507 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672164 6507 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672168 6507 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672172 6507 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672174 6507 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672177 6507 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672180 6507 flags.go:64] FLAG: --enable-server="true" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672183 6507 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672188 6507 flags.go:64] FLAG: --event-burst="100" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672191 6507 flags.go:64] FLAG: --event-qps="50" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672194 6507 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672197 6507 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672200 6507 flags.go:64] FLAG: --eviction-hard="" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672205 6507 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 23:13:50.678023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672208 6507 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672212 6507 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672216 6507 flags.go:64] FLAG: --eviction-soft="" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672219 6507 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672223 6507 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672227 6507 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672230 6507 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672233 6507 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672236 6507 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672239 6507 flags.go:64] FLAG: --feature-gates="" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672243 6507 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672246 6507 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672249 6507 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672253 6507 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672255 6507 flags.go:64] FLAG: --healthz-port="10248" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672259 6507 flags.go:64] FLAG: --help="false" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672262 6507 flags.go:64] FLAG: --hostname-override="ip-10-0-133-100.ec2.internal" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672266 6507 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672269 6507 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672272 6507 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672275 6507 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672278 6507 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672281 6507 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 23:13:50.678600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672284 6507 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672287 6507 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672290 6507 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672293 6507 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672296 6507 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672299 6507 flags.go:64] FLAG: --kube-reserved="" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672302 6507 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672305 6507 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672308 6507 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672310 6507 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672314 6507 flags.go:64] FLAG: --lock-file="" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672318 6507 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672321 6507 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672324 6507 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672331 6507 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672334 6507 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672337 6507 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672340 6507 flags.go:64] FLAG: --logging-format="text" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672343 6507 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672347 6507 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672350 6507 flags.go:64] FLAG: --manifest-url="" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672353 6507 flags.go:64] FLAG: --manifest-url-header="" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672357 6507 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672360 6507 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672364 6507 flags.go:64] FLAG: --max-pods="110" Oct 29 23:13:50.679199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672368 6507 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672372 6507 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672375 6507 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672378 6507 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672381 6507 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672383 6507 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672387 6507 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672394 6507 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672397 6507 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672400 6507 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672403 6507 flags.go:64] FLAG: --pod-cidr="" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672406 6507 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672409 6507 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672412 6507 flags.go:64] FLAG: --pods-per-core="0" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672415 6507 flags.go:64] FLAG: --port="10250" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672418 6507 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672421 6507 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03d251752a696b359" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672424 6507 flags.go:64] FLAG: --qos-reserved="" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672428 6507 flags.go:64] FLAG: --read-only-port="10255" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672431 6507 flags.go:64] FLAG: --register-node="true" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672434 6507 flags.go:64] FLAG: --register-with-taints="" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672440 6507 flags.go:64] FLAG: --registry-burst="10" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672443 6507 flags.go:64] FLAG: --registry-qps="5" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672446 6507 flags.go:64] FLAG: --reserved-cpus="" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672449 6507 flags.go:64] FLAG: --reserved-memory="" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672466 6507 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 23:13:50.679818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672471 6507 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672475 6507 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672478 6507 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672481 6507 flags.go:64] FLAG: --runonce="false" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672484 6507 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672487 6507 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672490 6507 flags.go:64] FLAG: --seccomp-default="false" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672494 6507 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672497 6507 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672500 6507 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672503 6507 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672506 6507 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672509 6507 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672512 6507 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672515 6507 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672518 6507 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672521 6507 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672524 6507 flags.go:64] FLAG: --system-cgroups="" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672527 6507 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672532 6507 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672535 6507 flags.go:64] FLAG: --tls-cert-file="" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672538 6507 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672542 6507 flags.go:64] FLAG: --tls-min-version="" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672545 6507 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672549 6507 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 23:13:50.680431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672552 6507 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672555 6507 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672560 6507 flags.go:64] FLAG: --v="2" Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672565 6507 flags.go:64] FLAG: --version="false" Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672569 6507 flags.go:64] FLAG: --vmodule="" Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672573 6507 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.672576 6507 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672751 6507 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672755 6507 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672758 6507 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672761 6507 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672764 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672766 6507 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672769 6507 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672772 6507 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672782 6507 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672785 6507 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672788 6507 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672791 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672793 6507 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672796 6507 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 23:13:50.681060 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672799 6507 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672802 6507 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672804 6507 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672807 6507 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672809 6507 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672812 6507 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672815 6507 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672817 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672820 6507 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672822 6507 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672827 6507 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672829 6507 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672832 6507 feature_gate.go:386] unrecognized feature gate: Example Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672836 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672838 6507 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672841 6507 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672844 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672846 6507 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672849 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672851 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 23:13:50.681659 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672854 6507 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672857 6507 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672859 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672862 6507 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672865 6507 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672868 6507 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672873 6507 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672883 6507 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672887 6507 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672890 6507 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672892 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672895 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672897 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672900 6507 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672903 6507 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672905 6507 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672908 6507 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672912 6507 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672915 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 23:13:50.682317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672918 6507 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672921 6507 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672923 6507 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672927 6507 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672930 6507 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672933 6507 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672937 6507 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672939 6507 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672942 6507 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672945 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672947 6507 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672950 6507 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672952 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672955 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672957 6507 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672960 6507 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672963 6507 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672965 6507 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672968 6507 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 23:13:50.683112 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672970 6507 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672973 6507 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672981 6507 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672984 6507 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672987 6507 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672989 6507 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672992 6507 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672995 6507 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.672997 6507 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673000 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673002 6507 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673005 6507 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673008 6507 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673010 6507 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673013 6507 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673015 6507 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673020 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673022 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673025 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673029 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 23:13:50.683877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673031 6507 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 23:13:50.684391 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673034 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 23:13:50.684391 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673037 6507 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 23:13:50.684391 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673040 6507 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 23:13:50.684391 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673042 6507 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 23:13:50.684391 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673045 6507 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 23:13:50.684391 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673047 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 23:13:50.684391 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673049 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 23:13:50.684391 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673052 6507 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 23:13:50.684391 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673055 6507 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 23:13:50.684391 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673057 6507 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 23:13:50.684391 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673060 6507 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 23:13:50.684391 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673063 6507 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 23:13:50.684391 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673065 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 23:13:50.684391 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.673090 6507 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 23:13:50.684391 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673131 6507 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 23:13:50.684391 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673137 6507 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673140 6507 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673143 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673146 6507 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673149 6507 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673151 6507 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673154 6507 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673157 6507 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673159 6507 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673162 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673164 6507 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673167 6507 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673170 6507 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673173 6507 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673176 6507 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673180 6507 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673182 6507 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673185 6507 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673187 6507 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673190 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 23:13:50.684919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673192 6507 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673195 6507 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673197 6507 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673200 6507 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673203 6507 feature_gate.go:386] unrecognized feature gate: Example Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673205 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673208 6507 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673210 6507 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673213 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673215 6507 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673218 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673220 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673226 6507 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673229 6507 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673231 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673234 6507 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673237 6507 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673241 6507 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673244 6507 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673247 6507 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 23:13:50.685634 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673250 6507 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673253 6507 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673255 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673258 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673260 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673263 6507 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673265 6507 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673269 6507 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673271 6507 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673275 6507 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673279 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673283 6507 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673286 6507 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673294 6507 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673297 6507 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673300 6507 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673303 6507 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673306 6507 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673308 6507 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 23:13:50.686125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673311 6507 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673313 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673316 6507 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673318 6507 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673321 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673323 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673326 6507 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673329 6507 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673332 6507 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673334 6507 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673337 6507 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673339 6507 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673342 6507 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673344 6507 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673347 6507 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673349 6507 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673352 6507 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673355 6507 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673358 6507 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673360 6507 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 23:13:50.687156 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673365 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673367 6507 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673370 6507 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673373 6507 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673375 6507 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673378 6507 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673380 6507 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673383 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673385 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673388 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673390 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673393 6507 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673396 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673399 6507 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673401 6507 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673404 6507 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673406 6507 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673409 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673411 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 23:13:50.687740 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673414 6507 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673418 6507 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673421 6507 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673425 6507 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673427 6507 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673430 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.673432 6507 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.677122 6507 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.677137 6507 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.677155 6507 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.677161 6507 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677228 6507 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677234 6507 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677238 6507 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677241 6507 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677243 6507 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677247 6507 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677250 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677253 6507 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 23:13:50.688191 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677255 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677258 6507 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677261 6507 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677264 6507 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677266 6507 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677269 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677272 6507 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677275 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677277 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677280 6507 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677283 6507 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677285 6507 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677288 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677292 6507 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677295 6507 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677298 6507 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677301 6507 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677304 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677306 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 23:13:50.688679 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677309 6507 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677312 6507 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677314 6507 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677317 6507 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677320 6507 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677323 6507 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677326 6507 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677328 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677331 6507 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677334 6507 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677336 6507 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677339 6507 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677341 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677344 6507 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677348 6507 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677351 6507 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677354 6507 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677357 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677360 6507 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677362 6507 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 23:13:50.689136 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677365 6507 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677368 6507 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677370 6507 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677373 6507 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677375 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677378 6507 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677381 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677385 6507 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677388 6507 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677390 6507 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677393 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677396 6507 feature_gate.go:386] unrecognized feature gate: Example Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677398 6507 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677401 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677403 6507 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677406 6507 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677409 6507 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677412 6507 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677414 6507 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677417 6507 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 23:13:50.689711 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677419 6507 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677422 6507 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677425 6507 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677429 6507 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677432 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677435 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677438 6507 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677441 6507 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677444 6507 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677447 6507 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677449 6507 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677452 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677470 6507 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677474 6507 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677478 6507 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677481 6507 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677484 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677486 6507 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677490 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 23:13:50.690181 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677492 6507 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677496 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677499 6507 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677501 6507 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677504 6507 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677506 6507 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677509 6507 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677511 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677514 6507 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677517 6507 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677519 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677522 6507 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677524 6507 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677527 6507 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677530 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677533 6507 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677535 6507 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677538 6507 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677540 6507 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 23:13:50.690731 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677543 6507 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 23:13:50.691184 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.677571 6507 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 23:13:50.691184 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677706 6507 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 23:13:50.691184 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677710 6507 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 23:13:50.691184 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677713 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 23:13:50.691184 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677716 6507 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 23:13:50.691184 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677718 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 23:13:50.691184 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677721 6507 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 23:13:50.691184 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677724 6507 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 23:13:50.691184 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677726 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 23:13:50.691184 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677729 6507 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 23:13:50.691184 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677732 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 23:13:50.691184 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677735 6507 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 23:13:50.691184 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677737 6507 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 23:13:50.691184 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677740 6507 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 23:13:50.691184 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677743 6507 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 23:13:50.691184 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677746 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677749 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677752 6507 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677754 6507 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677757 6507 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677759 6507 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677762 6507 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677764 6507 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677767 6507 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677769 6507 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677772 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677774 6507 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677777 6507 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677779 6507 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677782 6507 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677784 6507 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677788 6507 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677793 6507 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677796 6507 feature_gate.go:386] unrecognized feature gate: Example Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677798 6507 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 23:13:50.691698 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677801 6507 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677804 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677806 6507 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677809 6507 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677811 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677814 6507 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677816 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677819 6507 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677822 6507 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677825 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677827 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677830 6507 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677833 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677835 6507 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677838 6507 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677840 6507 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677842 6507 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677845 6507 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677847 6507 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677850 6507 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 23:13:50.692418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677853 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677855 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677858 6507 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677860 6507 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677863 6507 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677865 6507 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677868 6507 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677870 6507 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677872 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677875 6507 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677877 6507 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677884 6507 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677887 6507 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677890 6507 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677892 6507 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677895 6507 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677897 6507 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677900 6507 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677903 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677905 6507 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 23:13:50.692919 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677908 6507 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677910 6507 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677913 6507 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677916 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677919 6507 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677924 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677927 6507 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677930 6507 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677933 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677935 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677938 6507 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677940 6507 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677943 6507 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677946 6507 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677948 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677951 6507 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677953 6507 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677956 6507 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677958 6507 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 23:13:50.693400 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677961 6507 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 23:13:50.693882 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677963 6507 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 23:13:50.693882 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677966 6507 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 23:13:50.693882 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677969 6507 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 23:13:50.693882 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677973 6507 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 23:13:50.693882 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677976 6507 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 23:13:50.693882 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677979 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 23:13:50.693882 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677981 6507 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 23:13:50.693882 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677984 6507 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 23:13:50.693882 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677987 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 23:13:50.693882 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677989 6507 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 23:13:50.693882 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677992 6507 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 23:13:50.693882 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.677994 6507 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 23:13:50.693882 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.678017 6507 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 23:13:50.693882 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678054 6507 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 23:13:50.693882 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678059 6507 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 23:13:50.693882 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678062 6507 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678065 6507 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678068 6507 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678072 6507 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678074 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678077 6507 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678080 6507 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678082 6507 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678085 6507 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678088 6507 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678091 6507 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678095 6507 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678099 6507 feature_gate.go:386] unrecognized feature gate: Example Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678102 6507 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678105 6507 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678107 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678110 6507 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678113 6507 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678115 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 23:13:50.694265 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678118 6507 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678120 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678125 6507 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678127 6507 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678130 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678132 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678135 6507 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678138 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678140 6507 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678143 6507 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678145 6507 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678148 6507 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678150 6507 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678153 6507 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678155 6507 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678158 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678161 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678164 6507 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678167 6507 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678169 6507 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 23:13:50.694877 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678172 6507 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678175 6507 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678177 6507 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678181 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678183 6507 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678186 6507 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678188 6507 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678191 6507 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678193 6507 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678196 6507 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678198 6507 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678201 6507 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678203 6507 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678206 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678210 6507 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678213 6507 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678215 6507 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678218 6507 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678220 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 23:13:50.695443 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678224 6507 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678228 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678230 6507 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678233 6507 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678235 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678238 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678241 6507 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678243 6507 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678247 6507 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678249 6507 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678251 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678255 6507 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678258 6507 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678261 6507 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678263 6507 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678266 6507 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678268 6507 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678271 6507 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678274 6507 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 23:13:50.696002 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678276 6507 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678279 6507 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678281 6507 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678284 6507 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678286 6507 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678289 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678291 6507 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678294 6507 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678298 6507 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678300 6507 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678303 6507 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678306 6507 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678308 6507 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678311 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678314 6507 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678316 6507 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678319 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678321 6507 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678324 6507 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 23:13:50.696654 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678327 6507 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 23:13:50.697115 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678329 6507 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 23:13:50.697115 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678332 6507 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 23:13:50.697115 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678334 6507 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 23:13:50.697115 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678337 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 23:13:50.697115 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678340 6507 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 23:13:50.697115 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678343 6507 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 23:13:50.697115 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:13:50.678346 6507 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 23:13:50.697115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.678394 6507 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 23:13:50.697115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.679346 6507 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 29 23:13:50.697115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.679416 6507 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 29 23:13:50.697115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.679643 6507 server.go:1014] "Starting client certificate rotation" Oct 29 23:13:50.697115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.679788 6507 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 23:13:50.697115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.679842 6507 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 23:07:38 +0000 UTC" deadline="2028-05-12 01:46:05.889556711 +0000 UTC" Oct 29 23:13:50.697115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.679877 6507 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13442h32m15.20968242s" Oct 29 23:13:50.697115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.680150 6507 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 23:13:50.697522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.681301 6507 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 23:13:50.697522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.684436 6507 log.go:25] "Validated CRI v1 runtime API" Oct 29 23:13:50.697522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.686448 6507 log.go:25] "Validated CRI v1 image API" Oct 29 23:13:50.697522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.687202 6507 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 23:13:50.697522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.689315 6507 fs.go:135] Filesystem UUIDs: map[704d1010-6224-4aee-9bb5-357fc0ba23f7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7b96e30c-11d1-4091-ae3c-9d8dc4d51008:/dev/nvme0n1p3] Oct 29 23:13:50.697701 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.689334 6507 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01c61799f0cefae338425ed766986726baf5fdb2c6f21da29ea96e925f83766c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01c61799f0cefae338425ed766986726baf5fdb2c6f21da29ea96e925f83766c/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e95efb7515cff6c8da04318ad7bd15f28b23b54e978ce0d7ffba071236296d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e95efb7515cff6c8da04318ad7bd15f28b23b54e978ce0d7ffba071236296d0/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e016aca5752d2009a16b9120e9fecfd95d1f88efd55a16131bacab7721de5f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e016aca5752d2009a16b9120e9fecfd95d1f88efd55a16131bacab7721de5f6/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32883262f69fb005c5fba90b36f47abc0e94649e057d04cc200e2065222ee7db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32883262f69fb005c5fba90b36f47abc0e94649e057d04cc200e2065222ee7db/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56b460e0df4627dca42311dd173e6288fa2566a4c7b3ad1cfb32b6aa348178c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56b460e0df4627dca42311dd173e6288fa2566a4c7b3ad1cfb32b6aa348178c4/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/594c86e5a3b9bd1d16f3ecd79f6e62d63636c7e4b7add4c127d1758db64d2f45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/594c86e5a3b9bd1d16f3ecd79f6e62d63636c7e4b7add4c127d1758db64d2f45/userdata/shm major:0 minor:262 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66c61446fb25e53f3d0ae67b70a4dbd35e26e55bdc8fba9858feb4cd843c4d7b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66c61446fb25e53f3d0ae67b70a4dbd35e26e55bdc8fba9858feb4cd843c4d7b/userdata/shm major:0 minor:361 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69db1b82499ae56bbb1741cadf9f71338ce3ed100baf3ca2b0499152ccd7f52c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69db1b82499ae56bbb1741cadf9f71338ce3ed100baf3ca2b0499152ccd7f52c/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b338b2473e9686cee8d57ff28c46cef29f826b8c5b63dfee498236169935e16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b338b2473e9686cee8d57ff28c46cef29f826b8c5b63dfee498236169935e16/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f56629e6314704d696b28a21f2fbf244aebd8c6100286458fcd858c4d37d12c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f56629e6314704d696b28a21f2fbf244aebd8c6100286458fcd858c4d37d12c/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7329f3e20c8f7d066f85974cb05a3ae14853b884989d9c551ac11aa1fb7d232c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7329f3e20c8f7d066f85974cb05a3ae14853b884989d9c551ac11aa1fb7d232c/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b59a50e31096da6b08d11a397229750794b1e6af3c9959372a89b4288cd992d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b59a50e31096da6b08d11a397229750794b1e6af3c9959372a89b4288cd992d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/889f0098da894fab68152e20bf4a1979512d78ab550794b83e701dba7de9ffc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/889f0098da894fab68152e20bf4a1979512d78ab550794b83e701dba7de9ffc8/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a60a7805f22a1f6866107dc1e940782d727d1953974dc2eb1f1b264591047daf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a60a7805f22a1f6866107dc1e940782d727d1953974dc2eb1f1b264591047daf/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/adb4703c15ab556bbd5cd4bd0dbf1b92872238ccfb62730d4009dd77804760a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/adb4703c15ab556bbd5cd4bd0dbf1b92872238ccfb62730d4009dd77804760a6/userdata/shm major:0 minor:394 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b88d86003c8c53f05e764fb2a4528ce7149004260b8b362e64002cafc88871d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b88d86003c8c53f05e764fb2a4528ce7149004260b8b362e64002cafc88871d4/userdata/shm major:0 minor:391 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ccc0103ba9931583242315b5593b1382f02a9686b020724476322573c08170fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ccc0103ba9931583242315b5593b1382f02a9686b020724476322573c08170fa/userdata/shm major:0 minor:390 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/daaf958f300d0ddc0f2ccd8db050c5938b2f8f5eefc6781a84ee4100ad30380a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/daaf958f300d0ddc0f2ccd8db050c5938b2f8f5eefc6781a84ee4100ad30380a/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb5fcf7da1f60dc265c4cec8b0771d99a3915acbb6d6488ee5bb7fb65298f379/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb5fcf7da1f60dc265c4cec8b0771d99a3915acbb6d6488ee5bb7fb65298f379/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebc69002251bf669a9de3554274ef6e438737d19545670996eb1105f888e93a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebc69002251bf669a9de3554274ef6e438737d19545670996eb1105f888e93a7/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f47734bfb9a7143130d8cb58a4a3e3999c5b8a5ed3b82f5ae191b1c8babc1178/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f47734bfb9a7143130d8cb58a4a3e3999c5b8a5ed3b82f5ae191b1c8babc1178/userdata/shm major:0 minor:377 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fbe5b256b4f104384b21ffe6e16f9f413a15378154a8980529b4044551b908a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fbe5b256b4f104384b21ffe6e16f9f413a15378154a8980529b4044551b908a4/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc/volumes/kubernetes.io~projected/kube-api-access-56n47:{mountpoint:/var/lib/kubelet/pods/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc/volumes/kubernetes.io~projected/kube-api-access-56n47 major:0 minor:389 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b21e162-a2f3-4505-9dff-34a47e20d3e6/volumes/kubernetes.io~projected/kube-api-access-hvnmz:{mountpoint:/var/lib/kubelet/pods/1b21e162-a2f3-4505-9dff-34a47e20d3e6/volumes/kubernetes.io~projected/kube-api-access-hvnmz major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/254e83a6-ab81-4b6b-b5ce-0538ccaede49/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/254e83a6-ab81-4b6b-b5ce-0538ccaede49/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c3e5ae7-809c-475e-9623-d4d96e6da95c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2c3e5ae7-809c-475e-9623-d4d96e6da95c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c3e5ae7-809c-475e-9623-d4d96e6da95c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2c3e5ae7-809c-475e-9623-d4d96e6da95c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c3e5ae7-809c-475e-9623-d4d96e6da95c/volumes/kubernetes.io~projected/kube-api-access-jm9rx:{mountpoint:/var/lib/kubelet/pods/2c3e5ae7-809c-475e-9623-d4d96e6da95c/volumes/kubernetes.io~projected/kube-api-access-jm9rx major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34499222-f746-4e87-8dc8-54e6a25c75d6/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/34499222-f746-4e87-8dc8-54e6a25c75d6/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34499222-f746-4e87-8dc8-54e6a25c75d6/volumes/kubernetes.io~projected/kube-api-access-4kw4s:{mountpoint:/var/lib/kubelet/pods/34499222-f746-4e87-8dc8-54e6a25c75d6/volumes/kubernetes.io~projected/kube-api-access-4kw4s major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34499222-f746-4e87-8dc8-54e6a25c75d6/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/34499222-f746-4e87-8dc8-54e6a25c75d6/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volume-subpaths/web-config/config-reloader/2 major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volume-subpaths/web-config/prometheus/15 major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~empty-dir/config-out major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~projected/kube-api-access-6cxhf:{mountpoint:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~projected/kube-api-access-6cxhf major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~projected/tls-assets major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/config major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/web-config major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54576e13-0668-4255-952b-46b2bf5a279c/volumes/kubernetes.io~projected/kube-api-access-j8zs9:{mountpoint:/var/lib/kubelet/pods/54576e13-0668-4255-952b-46b2bf5a279c/volumes/kubernetes.io~projected/kube-api-access-j8zs9 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54576e13-0668-4255-952b-46b2bf5a279c/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/54576e13-0668-4255-952b-46b2bf5a279c/volumes/kubernetes.io~secret/metrics-certs major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6170bf16-e81d-40fd-a4a0-449213b94d19/volumes/kubernetes.io~projected/kube-api-access-zv2xt:{mountpoint:/var/lib/kubelet/pods/6170bf16-e81d-40fd-a4a0-449213b94d19/volumes/kubernetes.io~projected/kube-api-access-zv2xt major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volume-subpaths/web-config/alertmanager/9 major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volume-subpaths/web-config/config-reloader/6 major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~empty-dir/config-out major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~projected/kube-api-access-tnbnn:{mountpoint:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~projected/kube-api-access-tnbnn major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~projected/tls-assets major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/config-volume major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/web-config major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73623961-3ffb-4492-a2f4-74a5c4b9551e/volumes/kubernetes.io~projected/kube-api-access-np8k4:{mountpoint:/var/lib/kubelet/pods/73623961-3ffb-4492-a2f4-74a5c4b9551e/volumes/kubernetes.io~projected/kube-api-access-np8k4 major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73623961-3ffb-4492-a2f4-74a5c4b9551e/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/73623961-3ffb-4492-a2f4-74a5c4b9551e/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73623961-3ffb-4492-a2f4-74a5c4b9551e/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/73623961-3ffb-4492-a2f4-74a5c4b9551e/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76e09503-45e8-4e94-9a3b-a5d291a3990f/volumes/kubernetes.io~projected/kube-api-access-vr5kp:{mountpoint:/var/lib/kubelet/pods/76e09503-45e8-4e94-9a3b-a5d291a3990f/volumes/kubernetes.io~projected/kube-api-access-vr5kp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a00edc7-9624-4236-bb7c-9ac71e911416/volumes/kubernetes.io~projected/kube-api-access-9k4q5:{mountpoint:/var/lib/kubelet/pods/7a00edc7-9624-4236-bb7c-9ac71e911416/volumes/kubernetes.io~projected/kube-api-access-9k4q5 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a00edc7-9624-4236-bb7c-9ac71e911416/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/7a00edc7-9624-4236-bb7c-9ac71e911416/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e3f3838-8917-45a1-a496-652392caa72c/volumes/kubernetes.io~projected/kube-api-access-v6flh:{mountpoint:/var/lib/kubelet/pods/7e3f3838-8917-45a1-a496-652392caa72c/volumes/kubernetes.io~projected/kube-api-access-v6flh major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e3f3838-8917-45a1-a496-652392caa72c/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/7e3f3838-8917-45a1-a496-652392caa72c/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e3f3838-8917-45a1-a496-652392caa72c/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/7e3f3838-8917-45a1-a496-652392caa72c/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85acd560-02f0-4df7-b672-f0c099281e7a/volumes/kubernetes.io~projected/kube-api-access-4qspl:{mountpoint:/var/lib/kubelet/pods/85acd560-02f0-4df7-b672-f0c099281e7a/volumes/kubernetes.io~projected/kube-api-access-4qspl major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85acd560-02f0-4df7-b672-f0c099281e7a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/85acd560-02f0-4df7-b672-f0c099281e7a/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9/volumes/kubernetes.io~projected/kube-api-access-wc84k:{mountpoint:/var/lib/kubelet/pods/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9/volumes/kubernetes.io~projected/kube-api-access-wc84k major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cccc782-d044-4032-ba8b-8f4d4cdab41b/volumes/kubernetes.io~projected/kube-api-access-cxt65:{mountpoint:/var/lib/kubelet/pods/8cccc782-d044-4032-ba8b-8f4d4cdab41b/volumes/kubernetes.io~projected/kube-api-access-cxt65 major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cccc782-d044-4032-ba8b-8f4d4cdab41b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8cccc782-d044-4032-ba8b-8f4d4cdab41b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cccc782-d044-4032-ba8b-8f4d4cdab41b/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/8cccc782-d044-4032-ba8b-8f4d4cdab41b/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/916cb471-df04-4560-aaea-7e9d27f41897/volumes/kubernetes.io~projected/kube-api-access-858x5:{mountpoint:/var/lib/kubelet/pods/916cb471-df04-4560-aaea-7e9d27f41897/volumes/kubernetes.io~projected/kube-api-access-858x5 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/916cb471-df04-4560-aaea-7e9d27f41897/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/916cb471-df04-4560-aaea-7e9d27f41897/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcec0d7f-0330-434c-824a-246928597652/volumes/kubernetes.io~projected/kube-api-access-mg42k:{mountpoint:/var/lib/kubelet/pods/bcec0d7f-0330-434c-824a-246928597652/volumes/kubernetes.io~projected/kube-api-access-mg42k major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/defafe8e-3565-464e-b2be-522d526248a4/volumes/kubernetes.io~projected/kube-api-access-jnbnk:{mountpoint:/var/lib/kubelet/pods/defafe8e-3565-464e-b2be-522d526248a4/volumes/kubernetes.io~projected/kube-api-access-jnbnk major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/defafe8e-3565-464e-b2be-522d526248a4/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/defafe8e-3565-464e-b2be-522d526248a4/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/defafe8e-3565-464e-b2be-522d526248a4/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/defafe8e-3565-464e-b2be-522d526248a4/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6cba05f-3979-445f-a517-da054562cc20/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e6cba05f-3979-445f-a517-da054562cc20/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4a78746-396b-46fd-a42a-d819d866a6af/volumes/kubernetes.io~projected/kube-api-access-2l78l:{mountpoint:/var/lib/kubelet/pods/f4a78746-396b-46fd-a42a-d819d866a6af/volumes/kubernetes.io~projected/kube-api-access-2l78l major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/09c99182184c13fa1a7b0287e1cfa52eeeeffbb1f7496615012328c044252dae/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6d2fc981e14f3c95abc83337ef4f8f08e3784fea34dc5188c5a4067b8e51bdbc/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/68e963d4a1e803ed16c774b1b71599af5e8bd76b27f943e804c18580a5a0cca3/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7646718f4279026cdbbbfc8c82dc9491d1288ff62a1507a1d6ea34bf45c958e0/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/88c89947ea0d6dcb5143fd6c4bf37f6cd761cc729ce1b242653d661e6df16072/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b1aa5fd3c5ff85ef9b19c9007be7e82d0c2b5f1593716d9275e768352e567edf/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/57fd3baeaa4c4da33380da99db74f6af21b4878d5119710cccbf03c11f00d1dd/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6bfb24127492d133be2d9893f6d9be2d3814b3744f930678076f14c7f1096abb/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4a0aa05636881ef6ed7a6313d0b7e91de4895680ebe2216fb7a61e7e192bf70f/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/87772adea6ee51e3023362447562089dcaef3088f7745699d98dd8248ed289e0/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/d0c4626903f84c6891d22a02b1afcd98fb447e54267e4d17b2dd76a4bd99a479/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/cbeedf2ae3e1f614c5220105528badd4ea216c696259cac9651eb2ae1a8bb903/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6b1348080f474661f399b19789136e8f80281dfb0ad4c2e21f32d13933ec4371/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b785a0d4ad949c7eb2e1d23098a187f117d25509e94fd4a84881d830d84c8040/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/305ab9aadcaf2b523eb960cca5633ca0a444431b6f45374e2e2935630aecaf26/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/cd7b49220c208dd50e3173ff5967f09ce4351595b01661c58e6ed6467c1b446b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/df4fb973973096f97fa76c4a4fc3e784cea59eb3e7e6c5c8021b3ddeb7d9ba8f/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/25bbcf2c9b7be17cc5e41744164f9e6fef88e07057d81bf45c299f9a8d449728/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d39b53d78dcb86e6abef257ec050c7244d55f0024b33a52d2e37188f563dbb09/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b5761bbf035316373d29ebd0f76cdf055e9c30010125f0d2ce66ab517d8b5026/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/70b150f4e03740b957c55a3a814f0ecbc61f728db6de362b6eee85026c1da9b1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/31bcae600bc8ffbb6fed086a3fe82f72dadd0987929c786467f0df28491a9a03/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/0ed0e8cc6fa026c4670bc77837f1d127b36b6bd69ad1ca47de5785011a2c1b2d/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/5f2387ae27bdad423feb9cca27c6149a22f03c6ffcb6c859f4f57a06833a8315/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/10e5019746c604af48fc70b667d91324a7db17a0576717a7eb468388f997e614/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/b25ba30e17d37770280493860c20a46c451ab42d9ab178bb98dde5c5373c502c/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/46849fcf877f88a92523e787b37158974579d309779edb18d4e0e361b579a371/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/c4e1961facd2e1d5d4d0eb45202125ecf785436019d906f3464280d3cbe6b879/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/14d80f2370826ffaf9a50442e0210fe6122cd0f5c2c5f5f08be9bdac516983e9/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/37761068814fc2685a89e7fd1167689cee3acbea9dfabd4418926ae993c26ec5/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/2b855dd72372f28b0b587ffba0d410a5ea76ddf88ef298befd0a6a9628a57531/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/a0719ac7052d0557b7796fe1e0ed2df6eed9a101a1cb974aa4a413b5b90a8d69/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/e0f3a6a125e7db8e4ad3b4ec07dc13fdc875b02805734d8c3ef5007852a605d7/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/48766160a3cddc7f39e5d98b8926b4a86a780451321f1feeed22c80fd5be4954/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/ab257051874ae6af9190210061836322645f103effa77707924d12de89dbc78c/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/f4fbb3c288261f2c4b66d3f1ca5691f471cdc58f9106417dcc51beeabaf3bc39/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/a7155069ef02af182203f2fed6d07f28afd157557f63a289a0249c220f4876bb/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/d326d9891f06039e4f5f29ac12176978a6d2040167c8c31d258568c5ea854990/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/c6f27361f438379b8c20fa7c625aa732cc5001e13844a189c9979a0efea3d86a/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/7530ff42735f01f2b76f4a5f3ae5ca91a09f7735f5738c63e8650caa4a2ca12c/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/53f911293e6d9711bcfa0c163166d005881f7b83b450496d0a82d8ddce444686/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/2492994ce1e98c4583379c43f7508ced33d8c2680e24bd9a91e9e6dc816adca5/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/e1c1c83915fa1bf8154aa2cb2c3c61c38e0fbe44dcb867f36caa4968f250ab79/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/bf6fe1f056967be1b6ce33a8a238f22c72040d5a9c662edef251fd7b1766e6ea/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/806e9655cd0f9e421d263edf9548dd1f6331de0998a5e4b3e9370e432ef33cf2/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/4ec5a34cfd67a1c1af6f0a2a698f92c4a7632d5d3acfafd95e2e7abfdd406c38/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/76dcee07bd7bf9be728d7fcd2f2a2523515faee0b1c8fa4d1903260a0f791643/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/a019024e6739f7673dd6eafd996717ad276386c14163c26fbf0f23791c198aae/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/cd25373bf803e5f609345a7c675771f6f332083f63a34c01c12364503e0cf459/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/bb459ca7de317012321e7202789b28ae852fd41eb61ec3be78b10e086d65b2a5/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/f18fbd4576d875f02073ec30584dc19f72692edbece2ed973c412ef9ba63b671/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/1598083545dc7a826230b3c29d1de72dead6c60431037adbfb1d707a3c9e5989/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/7024ea23399b1db4c8361aa8e1100a5524b0e055c28ba7bd9d2a86e441cbe604/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/dc50df7b6e0e4eda785b38008f39ecdf5db41f3d5e649971c3a2ae6d1eebdb85/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/06251189120a7ba86ef63e68ac1ed09d23e3e85231bf9f17bf79430ce9dddf01/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/2b6e738483f54a8ac63995baae29f188b6eee311866a1bb193a954e728f4a1e3/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/7b709817659282b73658479297c4f8229a1b330b5cb22b625311381d940ee480/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/5e37f19a50983be7663ebb537dcc74416551ac568537945da527f3a807439b78/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/f15e483d4db5242a34fcfeffc806c56ed7bbc98910b28a20eae5bc4519bc9da3/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/b818fcaee2dc10f802d384b0aa5b0e2246760d35690c8d3e6c14eb295b7cebaf/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/6fd049f441a1064af0d53dc028890cab0f55ed72e3d80737d1635b08420c55b3/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/86890124ed97ca4392fed80c2878c90b88fdfb268bbab6d65d2607d7a982060f/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/e03ccd1b24ecf6cbd6e762c84b346472b47586edcb07f7a25aeade0f55540e92/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/30e55c137c004844bbb5ed6c374cace38784bb793b4a45ff4d4d7279926d24d3/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/61802d23762b268c79e23230bdbb5ffc3ebb1c7c048721c836373cd0431ebd16/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/763cdaf33691549eb2e794ec27bb993b6f2acad6354dba7def3810550b8eff4e/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/100cd47af3c63115aed4e97b4876ebf5a19af03a07e39549017426908da2a19a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/95177390fae62640d85b95d8380368cfa60b2541f9f0521248ba200b63538c67/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/b83161e7404c4942ff9ca6da57927936795192c5fa436b4903220fa5d6e1681c/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/b0c74c855fd7f15fcb3bb8668cf57b245bb16f7720b040e6784df893b4e4e208/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/b80238504cad49516cb7b4999add968a2e481dd59861a784f26bbc706ed65d53/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/4950d20af412ab2d8683b3c777a0a3944396039d61c013d8de261363e5c2a142/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/0301a2413e728ef53fc8a4927af50371d341a8f15052fce18efcb88f38e5b092/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/2f62a6e2b2027e86cea1b22458e281aea3f233268ce422dbb8eb80736e571d28/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 29 23:13:50.698119 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.697366 6507 manager.go:217] Machine: {Timestamp:2026-10-29 23:13:50.696864473 +0000 UTC m=+0.059869213 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3142658 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22f7b8284ee5bb1ac59f8855e8709b SystemUUID:ec22f7b8-284e-e5bb-1ac5-9f8855e8709b BootID:cb4ecad0-0bdb-4e9f-a9e1-abcfb0f4534e Filesystems:[{Device:/run/containers/storage/overlay-containers/ebc69002251bf669a9de3554274ef6e438737d19545670996eb1105f888e93a7/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8cccc782-d044-4032-ba8b-8f4d4cdab41b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:254 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66c61446fb25e53f3d0ae67b70a4dbd35e26e55bdc8fba9858feb4cd843c4d7b/userdata/shm DeviceMajor:0 DeviceMinor:361 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:374 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:375 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~projected/kube-api-access-tnbnn DeviceMajor:0 DeviceMinor:291 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c3e5ae7-809c-475e-9623-d4d96e6da95c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/34499222-f746-4e87-8dc8-54e6a25c75d6/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85acd560-02f0-4df7-b672-f0c099281e7a/volumes/kubernetes.io~projected/kube-api-access-4qspl DeviceMajor:0 DeviceMinor:202 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7e3f3838-8917-45a1-a496-652392caa72c/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:246 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6170bf16-e81d-40fd-a4a0-449213b94d19/volumes/kubernetes.io~projected/kube-api-access-zv2xt DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69db1b82499ae56bbb1741cadf9f71338ce3ed100baf3ca2b0499152ccd7f52c/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b338b2473e9686cee8d57ff28c46cef29f826b8c5b63dfee498236169935e16/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:288 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/54576e13-0668-4255-952b-46b2bf5a279c/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:383 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b21e162-a2f3-4505-9dff-34a47e20d3e6/volumes/kubernetes.io~projected/kube-api-access-hvnmz DeviceMajor:0 DeviceMinor:61 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:371 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:367 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/254e83a6-ab81-4b6b-b5ce-0538ccaede49/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:388 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9/volumes/kubernetes.io~projected/kube-api-access-wc84k DeviceMajor:0 DeviceMinor:60 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73623961-3ffb-4492-a2f4-74a5c4b9551e/volumes/kubernetes.io~projected/kube-api-access-np8k4 DeviceMajor:0 DeviceMinor:257 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:368 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b88d86003c8c53f05e764fb2a4528ce7149004260b8b362e64002cafc88871d4/userdata/shm DeviceMajor:0 DeviceMinor:391 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56b460e0df4627dca42311dd173e6288fa2566a4c7b3ad1cfb32b6aa348178c4/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34499222-f746-4e87-8dc8-54e6a25c75d6/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:284 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:285 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/916cb471-df04-4560-aaea-7e9d27f41897/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:366 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54576e13-0668-4255-952b-46b2bf5a279c/volumes/kubernetes.io~projected/kube-api-access-j8zs9 DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/defafe8e-3565-464e-b2be-522d526248a4/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:355 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8cccc782-d044-4032-ba8b-8f4d4cdab41b/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:256 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/daaf958f300d0ddc0f2ccd8db050c5938b2f8f5eefc6781a84ee4100ad30380a/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/594c86e5a3b9bd1d16f3ecd79f6e62d63636c7e4b7add4c127d1758db64d2f45/userdata/shm DeviceMajor:0 DeviceMinor:262 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:288 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:370 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:367 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01c61799f0cefae338425ed766986726baf5fdb2c6f21da29ea96e925f83766c/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32883262f69fb005c5fba90b36f47abc0e94649e057d04cc200e2065222ee7db/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc/volumes/kubernetes.io~projected/kube-api-access-56n47 DeviceMajor:0 DeviceMinor:389 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85acd560-02f0-4df7-b672-f0c099281e7a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c3e5ae7-809c-475e-9623-d4d96e6da95c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/adb4703c15ab556bbd5cd4bd0dbf1b92872238ccfb62730d4009dd77804760a6/userdata/shm DeviceMajor:0 DeviceMinor:394 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb5fcf7da1f60dc265c4cec8b0771d99a3915acbb6d6488ee5bb7fb65298f379/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:279 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/7a00edc7-9624-4236-bb7c-9ac71e911416/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:286 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:373 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/bcec0d7f-0330-434c-824a-246928597652/volumes/kubernetes.io~projected/kube-api-access-mg42k DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76e09503-45e8-4e94-9a3b-a5d291a3990f/volumes/kubernetes.io~projected/kube-api-access-vr5kp DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/34499222-f746-4e87-8dc8-54e6a25c75d6/volumes/kubernetes.io~projected/kube-api-access-4kw4s DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/defafe8e-3565-464e-b2be-522d526248a4/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:359 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:367 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c3e5ae7-809c-475e-9623-d4d96e6da95c/volumes/kubernetes.io~projected/kube-api-access-jm9rx DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e016aca5752d2009a16b9120e9fecfd95d1f88efd55a16131bacab7721de5f6/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/916cb471-df04-4560-aaea-7e9d27f41897/volumes/kubernetes.io~projected/kube-api-access-858x5 DeviceMajor:0 DeviceMinor:203 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7a00edc7-9624-4236-bb7c-9ac71e911416/volumes/kubernetes.io~projected/kube-api-access-9k4q5 DeviceMajor:0 DeviceMinor:204 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:289 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/defafe8e-3565-464e-b2be-522d526248a4/volumes/kubernetes.io~projected/kube-api-access-jnbnk DeviceMajor:0 DeviceMinor:360 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fbe5b256b4f104384b21ffe6e16f9f413a15378154a8980529b4044551b908a4/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e95efb7515cff6c8da04318ad7bd15f28b23b54e978ce0d7ffba071236296d0/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73623961-3ffb-4492-a2f4-74a5c4b9551e/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:253 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:283 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:369 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:287 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e3f3838-8917-45a1-a496-652392caa72c/volumes/kubernetes.io~projected/kube-api-access-v6flh DeviceMajor:0 DeviceMinor:252 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ccc0103ba9931583242315b5593b1382f02a9686b020724476322573c08170fa/userdata/shm DeviceMajor:0 DeviceMinor:390 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/f4a78746-396b-46fd-a42a-d819d866a6af/volumes/kubernetes.io~projected/kube-api-access-2l78l DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7329f3e20c8f7d066f85974cb05a3ae14853b884989d9c551ac11aa1fb7d232c/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f47734bfb9a7143130d8cb58a4a3e3999c5b8a5ed3b82f5ae191b1c8babc1178/userdata/shm DeviceMajor:0 DeviceMinor:377 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73623961-3ffb-4492-a2f4-74a5c4b9551e/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:255 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:365 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:372 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:367 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a60a7805f22a1f6866107dc1e940782d727d1953974dc2eb1f1b264591047daf/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7e3f3838-8917-45a1-a496-652392caa72c/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:251 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cccc782-d044-4032-ba8b-8f4d4cdab41b/volumes/kubernetes.io~projected/kube-api-access-cxt65 DeviceMajor:0 DeviceMinor:258 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/889f0098da894fab68152e20bf4a1979512d78ab550794b83e701dba7de9ffc8/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f56629e6314704d696b28a21f2fbf244aebd8c6100286458fcd858c4d37d12c/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:288 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b59a50e31096da6b08d11a397229750794b1e6af3c9959372a89b4288cd992d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes/kubernetes.io~projected/kube-api-access-6cxhf DeviceMajor:0 DeviceMinor:376 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:288 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:284 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e6cba05f-3979-445f-a517-da054562cc20/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:290 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:32883262f69fb00 MacAddress:32:cd:48:30:93:a4 Speed:10000 Mtu:8901} {Name:66c61446fb25e53 MacAddress:26:b9:a1:cb:0c:7e Speed:10000 Mtu:8901} {Name:69db1b82499ae56 MacAddress:5e:55:91:e8:19:4f Speed:10000 Mtu:8901} {Name:6b338b2473e9686 MacAddress:ce:d1:5a:3e:fb:78 Speed:10000 Mtu:8901} {Name:6f56629e6314704 MacAddress:c2:2a:d7:c1:fd:35 Speed:10000 Mtu:8901} {Name:7329f3e20c8f7d0 MacAddress:32:d3:18:d0:2d:18 Speed:10000 Mtu:8901} {Name:889f0098da894fa MacAddress:26:ae:87:4d:87:2f Speed:10000 Mtu:8901} {Name:adb4703c15ab556 MacAddress:9a:e1:22:0c:3b:4d Speed:10000 Mtu:8901} {Name:b88d86003c8c53f MacAddress:82:45:d2:8d:e7:b3 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ff:8a:47:83 Speed:0 Mtu:9001} {Name:br-int MacAddress:52:56:9d:44:78:6e Speed:0 Mtu:8901} {Name:ccc0103ba993158 MacAddress:9a:57:bf:c0:c1:67 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ff:8a:47:83 Speed:0 Mtu:9001} {Name:f47734bfb9a7143 MacAddress:b2:08:76:e7:75:d8 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:4e:9e:ce:fe:3f:43 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e2:f8:77:9d:f8:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 23:13:50.698119 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.697941 6507 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 23:13:50.698423 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.698121 6507 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 23:13:50.698423 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.698392 6507 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 23:13:50.698626 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.698411 6507 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-100.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 23:13:50.698626 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.698573 6507 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 23:13:50.698626 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.698581 6507 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 23:13:50.698626 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.698592 6507 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 23:13:50.698626 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.698611 6507 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 23:13:50.698860 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.698710 6507 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 23:13:50.698908 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.698896 6507 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 23:13:50.699037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.699025 6507 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 23:13:50.699078 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.699069 6507 kubelet.go:493] "Attempting to sync node with API server" Oct 29 23:13:50.699108 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.699087 6507 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 23:13:50.699108 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.699102 6507 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 23:13:50.699183 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.699114 6507 kubelet.go:405] "Adding apiserver pod source" Oct 29 23:13:50.699183 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.699125 6507 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 23:13:50.700092 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.700074 6507 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 23:13:50.700092 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.700087 6507 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 23:13:50.700713 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.700688 6507 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 23:13:50.704800 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.704303 6507 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 23:13:50.704957 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.704947 6507 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 23:13:50.705077 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705065 6507 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 29 23:13:50.705270 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705259 6507 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 23:13:50.705302 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705289 6507 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 23:13:50.705388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705379 6507 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 23:13:50.705424 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705393 6507 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 23:13:50.705424 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705400 6507 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 23:13:50.705424 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705406 6507 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 23:13:50.705424 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705412 6507 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 23:13:50.705424 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705417 6507 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 23:13:50.705424 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705423 6507 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 23:13:50.705647 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705429 6507 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 23:13:50.705647 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705436 6507 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 23:13:50.705647 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705442 6507 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 23:13:50.705647 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705450 6507 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 23:13:50.705647 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705477 6507 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 23:13:50.705647 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705501 6507 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 23:13:50.705647 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.705507 6507 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 23:13:50.708868 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.708852 6507 server.go:1263] "Started kubelet" Oct 29 23:13:50.712759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.709095 6507 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 23:13:50.712759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.709178 6507 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 23:13:50.712759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.709186 6507 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 23:13:50.712759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.709586 6507 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 23:13:50.712759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.711280 6507 server.go:317] "Adding debug handlers to kubelet server" Oct 29 23:13:50.712759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.712404 6507 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 23:13:50.712759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.712535 6507 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 23:13:50.712759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.712564 6507 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 23:13:50.712759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.712576 6507 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 23:13:50.712759 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:50.712593 6507 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-100.ec2.internal\" not found" Oct 29 23:13:50.712759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.712656 6507 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 23:13:50.709385 ip-10-0-133-100 systemd[1]: Started Kubernetes Kubelet. Oct 29 23:13:50.714210 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.714095 6507 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 23:07:40 +0000 UTC" deadline="2028-05-22 18:59:54.048639692 +0000 UTC" Oct 29 23:13:50.714210 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.714144 6507 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13699h46m3.334500585s" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.714989 6507 factory.go:153] Registering CRI-O factory Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.715014 6507 factory.go:223] Registration of the crio container factory successfully Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.715101 6507 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.715112 6507 factory.go:55] Registering systemd factory Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.715121 6507 factory.go:223] Registration of the systemd container factory successfully Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.715152 6507 factory.go:103] Registering Raw factory Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.715168 6507 manager.go:1199] Started watching for new ooms in manager Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.715564 6507 manager.go:319] Starting recovery of all containers Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716450 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b21e162-a2f3-4505-9dff-34a47e20d3e6" volumeName="kubernetes.io/configmap/1b21e162-a2f3-4505-9dff-34a47e20d3e6-iptables-alerter-script" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716507 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-metrics-client-ca" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716523 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716537 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716550 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-tls-assets" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716576 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e2d17bd-2440-419f-9403-73f10f421359" volumeName="kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-metrics-client-ca" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716592 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e2d17bd-2440-419f-9403-73f10f421359" volumeName="kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-config-out" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716605 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e2d17bd-2440-419f-9403-73f10f421359" volumeName="kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716621 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e2d17bd-2440-419f-9403-73f10f421359" volumeName="kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716634 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73623961-3ffb-4492-a2f4-74a5c4b9551e" volumeName="kubernetes.io/configmap/73623961-3ffb-4492-a2f4-74a5c4b9551e-metrics-client-ca" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716649 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716690 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c3e5ae7-809c-475e-9623-d4d96e6da95c" volumeName="kubernetes.io/projected/2c3e5ae7-809c-475e-9623-d4d96e6da95c-kube-api-access-jm9rx" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716720 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34499222-f746-4e87-8dc8-54e6a25c75d6" volumeName="kubernetes.io/secret/34499222-f746-4e87-8dc8-54e6a25c75d6-ovn-node-metrics-cert" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716734 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716751 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716764 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config-out" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716775 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-grpc-tls" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716789 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54576e13-0668-4255-952b-46b2bf5a279c" volumeName="kubernetes.io/projected/54576e13-0668-4255-952b-46b2bf5a279c-kube-api-access-j8zs9" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716807 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e2d17bd-2440-419f-9403-73f10f421359" volumeName="kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-config-volume" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716819 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e2d17bd-2440-419f-9403-73f10f421359" volumeName="kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716830 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cccc782-d044-4032-ba8b-8f4d4cdab41b" volumeName="kubernetes.io/projected/8cccc782-d044-4032-ba8b-8f4d4cdab41b-kube-api-access-cxt65" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716873 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="916cb471-df04-4560-aaea-7e9d27f41897" volumeName="kubernetes.io/empty-dir/916cb471-df04-4560-aaea-7e9d27f41897-tmp-dir" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716887 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e2d17bd-2440-419f-9403-73f10f421359" volumeName="kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-web-config" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716900 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85acd560-02f0-4df7-b672-f0c099281e7a" volumeName="kubernetes.io/projected/85acd560-02f0-4df7-b672-f0c099281e7a-kube-api-access-4qspl" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716914 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9" volumeName="kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cni-binary-copy" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716926 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="defafe8e-3565-464e-b2be-522d526248a4" volumeName="kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716939 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="defafe8e-3565-464e-b2be-522d526248a4" volumeName="kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716950 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="defafe8e-3565-464e-b2be-522d526248a4" volumeName="kubernetes.io/empty-dir/defafe8e-3565-464e-b2be-522d526248a4-tmp" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716967 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc" volumeName="kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716978 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c3e5ae7-809c-475e-9623-d4d96e6da95c" volumeName="kubernetes.io/empty-dir/2c3e5ae7-809c-475e-9623-d4d96e6da95c-tmp" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.716989 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34499222-f746-4e87-8dc8-54e6a25c75d6" volumeName="kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-ovnkube-script-lib" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717000 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73623961-3ffb-4492-a2f4-74a5c4b9551e" volumeName="kubernetes.io/secret/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717011 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a00edc7-9624-4236-bb7c-9ac71e911416" volumeName="kubernetes.io/projected/7a00edc7-9624-4236-bb7c-9ac71e911416-kube-api-access-9k4q5" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717022 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85acd560-02f0-4df7-b672-f0c099281e7a" volumeName="kubernetes.io/secret/85acd560-02f0-4df7-b672-f0c099281e7a-cert" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717033 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="defafe8e-3565-464e-b2be-522d526248a4" volumeName="kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717045 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="defafe8e-3565-464e-b2be-522d526248a4" volumeName="kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717059 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b21e162-a2f3-4505-9dff-34a47e20d3e6" volumeName="kubernetes.io/projected/1b21e162-a2f3-4505-9dff-34a47e20d3e6-kube-api-access-hvnmz" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717071 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-metrics-client-certs" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717082 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a00edc7-9624-4236-bb7c-9ac71e911416" volumeName="kubernetes.io/secret/7a00edc7-9624-4236-bb7c-9ac71e911416-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717094 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cccc782-d044-4032-ba8b-8f4d4cdab41b" volumeName="kubernetes.io/configmap/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717106 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34499222-f746-4e87-8dc8-54e6a25c75d6" volumeName="kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-env-overrides" seLinuxMountContext="" Oct 29 23:13:50.717168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717117 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34499222-f746-4e87-8dc8-54e6a25c75d6" volumeName="kubernetes.io/projected/34499222-f746-4e87-8dc8-54e6a25c75d6-kube-api-access-4kw4s" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717134 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717716 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6170bf16-e81d-40fd-a4a0-449213b94d19" volumeName="kubernetes.io/configmap/6170bf16-e81d-40fd-a4a0-449213b94d19-serviceca" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717730 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76e09503-45e8-4e94-9a3b-a5d291a3990f" volumeName="kubernetes.io/projected/76e09503-45e8-4e94-9a3b-a5d291a3990f-kube-api-access-vr5kp" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717795 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="916cb471-df04-4560-aaea-7e9d27f41897" volumeName="kubernetes.io/projected/916cb471-df04-4560-aaea-7e9d27f41897-kube-api-access-858x5" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717803 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c3e5ae7-809c-475e-9623-d4d96e6da95c" volumeName="kubernetes.io/empty-dir/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-tuned" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717817 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717826 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717836 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54576e13-0668-4255-952b-46b2bf5a279c" volumeName="kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717847 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e3f3838-8917-45a1-a496-652392caa72c" volumeName="kubernetes.io/projected/7e3f3838-8917-45a1-a496-652392caa72c-kube-api-access-v6flh" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717855 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="916cb471-df04-4560-aaea-7e9d27f41897" volumeName="kubernetes.io/configmap/916cb471-df04-4560-aaea-7e9d27f41897-config-volume" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717867 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73623961-3ffb-4492-a2f4-74a5c4b9551e" volumeName="kubernetes.io/projected/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-api-access-np8k4" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717875 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-kube-api-access-6cxhf" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717884 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9" volumeName="kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cni-sysctl-allowlist" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717895 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cccc782-d044-4032-ba8b-8f4d4cdab41b" volumeName="kubernetes.io/secret/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-tls" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717918 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6cba05f-3979-445f-a517-da054562cc20" volumeName="kubernetes.io/configmap/e6cba05f-3979-445f-a517-da054562cc20-konnectivity-ca" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717926 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717938 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cccc782-d044-4032-ba8b-8f4d4cdab41b" volumeName="kubernetes.io/secret/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717946 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-web-config" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717957 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717966 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e2d17bd-2440-419f-9403-73f10f421359" volumeName="kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.717973 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e2d17bd-2440-419f-9403-73f10f421359" volumeName="kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-tls-assets" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718016 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e3f3838-8917-45a1-a496-652392caa72c" volumeName="kubernetes.io/secret/7e3f3838-8917-45a1-a496-652392caa72c-openshift-state-metrics-tls" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718024 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6cba05f-3979-445f-a517-da054562cc20" volumeName="kubernetes.io/secret/e6cba05f-3979-445f-a517-da054562cc20-agent-certs" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718035 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="254e83a6-ab81-4b6b-b5ce-0538ccaede49" volumeName="kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718044 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-db" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718056 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76e09503-45e8-4e94-9a3b-a5d291a3990f" volumeName="kubernetes.io/empty-dir/76e09503-45e8-4e94-9a3b-a5d291a3990f-tmp-dir" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718148 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e3f3838-8917-45a1-a496-652392caa72c" volumeName="kubernetes.io/secret/7e3f3838-8917-45a1-a496-652392caa72c-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718157 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718181 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34499222-f746-4e87-8dc8-54e6a25c75d6" volumeName="kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-ovnkube-config" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718190 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" volumeName="kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718202 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6170bf16-e81d-40fd-a4a0-449213b94d19" volumeName="kubernetes.io/projected/6170bf16-e81d-40fd-a4a0-449213b94d19-kube-api-access-zv2xt" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718294 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e2d17bd-2440-419f-9403-73f10f421359" volumeName="kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718302 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e2d17bd-2440-419f-9403-73f10f421359" volumeName="kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-cluster-tls-config" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718315 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73623961-3ffb-4492-a2f4-74a5c4b9551e" volumeName="kubernetes.io/empty-dir/73623961-3ffb-4492-a2f4-74a5c4b9551e-volume-directive-shadow" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718325 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73623961-3ffb-4492-a2f4-74a5c4b9551e" volumeName="kubernetes.io/secret/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-tls" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718337 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73623961-3ffb-4492-a2f4-74a5c4b9551e" volumeName="kubernetes.io/configmap/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718346 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a00edc7-9624-4236-bb7c-9ac71e911416" volumeName="kubernetes.io/configmap/7a00edc7-9624-4236-bb7c-9ac71e911416-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718354 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9" volumeName="kubernetes.io/projected/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-kube-api-access-wc84k" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718410 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcec0d7f-0330-434c-824a-246928597652" volumeName="kubernetes.io/configmap/bcec0d7f-0330-434c-824a-246928597652-multus-daemon-config" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718422 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcec0d7f-0330-434c-824a-246928597652" volumeName="kubernetes.io/configmap/bcec0d7f-0330-434c-824a-246928597652-cni-binary-copy" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718514 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcec0d7f-0330-434c-824a-246928597652" volumeName="kubernetes.io/projected/bcec0d7f-0330-434c-824a-246928597652-kube-api-access-mg42k" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718527 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="defafe8e-3565-464e-b2be-522d526248a4" volumeName="kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718550 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="defafe8e-3565-464e-b2be-522d526248a4" volumeName="kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718566 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e2d17bd-2440-419f-9403-73f10f421359" volumeName="kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-main-db" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718576 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a00edc7-9624-4236-bb7c-9ac71e911416" volumeName="kubernetes.io/empty-dir/7a00edc7-9624-4236-bb7c-9ac71e911416-data-volume" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718593 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e3f3838-8917-45a1-a496-652392caa72c" volumeName="kubernetes.io/configmap/7e3f3838-8917-45a1-a496-652392caa72c-metrics-client-ca" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718605 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cccc782-d044-4032-ba8b-8f4d4cdab41b" volumeName="kubernetes.io/configmap/8cccc782-d044-4032-ba8b-8f4d4cdab41b-metrics-client-ca" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718638 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="defafe8e-3565-464e-b2be-522d526248a4" volumeName="kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718656 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4a78746-396b-46fd-a42a-d819d866a6af" volumeName="kubernetes.io/projected/f4a78746-396b-46fd-a42a-d819d866a6af-kube-api-access-2l78l" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718680 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e2d17bd-2440-419f-9403-73f10f421359" volumeName="kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-kube-api-access-tnbnn" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718699 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9" volumeName="kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718711 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cccc782-d044-4032-ba8b-8f4d4cdab41b" volumeName="kubernetes.io/empty-dir/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-textfile" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718736 6507 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="916cb471-df04-4560-aaea-7e9d27f41897" volumeName="kubernetes.io/secret/916cb471-df04-4560-aaea-7e9d27f41897-metrics-tls" seLinuxMountContext="" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718749 6507 reconstruct.go:97] "Volume reconstruction finished" Oct 29 23:13:50.719558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.718762 6507 reconciler.go:29] "Reconciler: start to sync state" Oct 29 23:13:50.743512 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.743433 6507 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 23:13:50.745087 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.745056 6507 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 23:13:50.745087 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.745092 6507 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 23:13:50.745285 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.745116 6507 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 23:13:50.745285 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:50.745157 6507 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 29 23:13:50.756657 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.756606 6507 generic.go:356] "Generic (PLEG): container finished" podID="85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9" containerID="9fa2d4d135e7d77c279f5286ed10bc14e0dd4bb298aa4441ab35a61983e42c1e" exitCode=0 Oct 29 23:13:50.756657 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.756654 6507 generic.go:356] "Generic (PLEG): container finished" podID="85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9" containerID="e7bebcffe8486b5a5930999d95310ee8ba35cde3dffbf832104a238a13fbc31d" exitCode=0 Oct 29 23:13:50.756838 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.756667 6507 generic.go:356] "Generic (PLEG): container finished" podID="85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9" containerID="2b444d4c00171b3d9f84c366d156428e4bd1d642b5bbdba9f0c331af1bee168b" exitCode=0 Oct 29 23:13:50.756838 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.756676 6507 generic.go:356] "Generic (PLEG): container finished" podID="85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9" containerID="70a43c38036d94f3eed1db52bf5fa1871ad9588bdcd14d3497e79bd3b277693f" exitCode=0 Oct 29 23:13:50.756838 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.756685 6507 generic.go:356] "Generic (PLEG): container finished" podID="85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9" containerID="85c59069ff6058e7716172c696cebd8963c11d097af46cbae05ae76c8d3f669b" exitCode=0 Oct 29 23:13:50.756838 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.756693 6507 generic.go:356] "Generic (PLEG): container finished" podID="85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9" containerID="c84d64b2d105cb404db4f430ab67ebd5325a98803d6d58da66648423301d346a" exitCode=0 Oct 29 23:13:50.760136 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.760109 6507 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 23:13:50.760580 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.760561 6507 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 29 23:13:50.761016 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.760989 6507 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 23:13:50.767817 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.767788 6507 generic.go:356] "Generic (PLEG): container finished" podID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerID="58271d5668a0a40f2a7a29fde78624521d20d0b178611b7075a8e004751862dc" exitCode=0 Oct 29 23:13:50.774391 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.774350 6507 generic.go:356] "Generic (PLEG): container finished" podID="6e2d17bd-2440-419f-9403-73f10f421359" containerID="eafbb4d1a4d3ec0c0d9881397c0f6acbc4799761dde36166a54b45df8fd53bf1" exitCode=0 Oct 29 23:13:50.781449 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.781427 6507 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 23:13:50.781777 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.781752 6507 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 23:13:50.785276 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.785252 6507 generic.go:356] "Generic (PLEG): container finished" podID="563071327f3cc335e24937880997d7f4" containerID="f8257adbd933d9107b5ce29dff2f2c66a2377748ffe4d75532599bb7fe7d6ac5" exitCode=0 Oct 29 23:13:50.794272 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.794244 6507 generic.go:356] "Generic (PLEG): container finished" podID="8cccc782-d044-4032-ba8b-8f4d4cdab41b" containerID="5dd671a85d692728e2fe38cd21b44ad5d0d124939559070f52276a9139154811" exitCode=0 Oct 29 23:13:50.830367 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.830283 6507 manager.go:324] Recovery completed Oct 29 23:13:50.846088 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:50.846050 6507 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 29 23:13:50.854994 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.854969 6507 cpu_manager.go:226] "Starting" policy="none" Oct 29 23:13:50.854994 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.854989 6507 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 23:13:50.855174 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.855009 6507 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 23:13:50.855174 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.855152 6507 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 29 23:13:50.855230 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.855165 6507 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 29 23:13:50.855230 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.855185 6507 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 29 23:13:50.855230 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.855191 6507 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 29 23:13:50.855230 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.855196 6507 policy_none.go:50] "Start" Oct 29 23:13:50.855230 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.855205 6507 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 23:13:50.855230 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.855217 6507 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 23:13:50.856313 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.856301 6507 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 29 23:13:50.856351 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.856314 6507 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 29 23:13:50.856351 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.856320 6507 policy_none.go:44] "Start" Oct 29 23:13:50.861054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.861026 6507 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 23:13:50.864623 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.864608 6507 manager.go:341] "Starting Device Plugin manager" Oct 29 23:13:50.865110 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:50.864639 6507 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 23:13:50.865110 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.865112 6507 server.go:87] "Starting device plugin registration server" Oct 29 23:13:50.865392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.865380 6507 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 23:13:50.865428 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.865394 6507 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 23:13:50.865515 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.865489 6507 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 23:13:50.865635 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.865600 6507 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 23:13:50.865729 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.865652 6507 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 23:13:50.865729 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.865663 6507 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 23:13:50.865822 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.865739 6507 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T23:13:50.865633817Z","UUID":"52971145-ed87-4a7d-9647-39e94e352c40","Handler":null,"Name":"","Endpoint":""} Oct 29 23:13:50.867629 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:50.867601 6507 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 23:13:50.868117 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.868098 6507 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 23:13:50.868204 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.868141 6507 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 23:13:50.966174 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.966135 6507 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 23:13:50.968173 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.968155 6507 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientMemory" Oct 29 23:13:50.968248 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.968195 6507 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 23:13:50.968248 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.968209 6507 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-100.ec2.internal" event="NodeHasSufficientPID" Oct 29 23:13:50.968320 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:50.968253 6507 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.002409 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.002379 6507 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.002557 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.002491 6507 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.046398 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.046342 6507 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal"] Oct 29 23:13:51.047091 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.047057 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" event={"ID":"563071327f3cc335e24937880997d7f4","Type":"ContainerStarted","Data":"aca5f43ea6f0601cf0150b76a5ac8c71a9ee7da51d5a99f81d78b856b605a295"} Oct 29 23:13:51.047133 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.047096 6507 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.047133 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.047100 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" event={"ID":"563071327f3cc335e24937880997d7f4","Type":"ContainerDied","Data":"f8257adbd933d9107b5ce29dff2f2c66a2377748ffe4d75532599bb7fe7d6ac5"} Oct 29 23:13:51.047133 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.047124 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" event={"ID":"563071327f3cc335e24937880997d7f4","Type":"ContainerStarted","Data":"eb5fcf7da1f60dc265c4cec8b0771d99a3915acbb6d6488ee5bb7fb65298f379"} Oct 29 23:13:51.047236 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.047137 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" event={"ID":"3a7e411141454709a32dcea46ca88fb7","Type":"ContainerStarted","Data":"49c8656f64c9cbedc34216c69440d1457dcd3607976a0547d4bd52b030bcf1ba"} Oct 29 23:13:51.047236 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.047149 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" event={"ID":"3a7e411141454709a32dcea46ca88fb7","Type":"ContainerStarted","Data":"fbe5b256b4f104384b21ffe6e16f9f413a15378154a8980529b4044551b908a4"} Oct 29 23:13:51.047236 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.047219 6507 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.070798 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.070767 6507 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 23:13:51.070918 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.070824 6507 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-100.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.071291 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.071276 6507 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 23:13:51.071340 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.071318 6507 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.162723 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.162649 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/563071327f3cc335e24937880997d7f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal\" (UID: \"563071327f3cc335e24937880997d7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.162723 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.162683 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/563071327f3cc335e24937880997d7f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal\" (UID: \"563071327f3cc335e24937880997d7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.162912 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.162706 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a7e411141454709a32dcea46ca88fb7-config\") pod \"kube-apiserver-proxy-ip-10-0-133-100.ec2.internal\" (UID: \"3a7e411141454709a32dcea46ca88fb7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.263289 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.263248 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/563071327f3cc335e24937880997d7f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal\" (UID: \"563071327f3cc335e24937880997d7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.263289 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.263287 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/563071327f3cc335e24937880997d7f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal\" (UID: \"563071327f3cc335e24937880997d7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.263498 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.263305 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a7e411141454709a32dcea46ca88fb7-config\") pod \"kube-apiserver-proxy-ip-10-0-133-100.ec2.internal\" (UID: \"3a7e411141454709a32dcea46ca88fb7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.263498 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.263344 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a7e411141454709a32dcea46ca88fb7-config\") pod \"kube-apiserver-proxy-ip-10-0-133-100.ec2.internal\" (UID: \"3a7e411141454709a32dcea46ca88fb7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.263498 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.263350 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/563071327f3cc335e24937880997d7f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal\" (UID: \"563071327f3cc335e24937880997d7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.263498 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.263356 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/563071327f3cc335e24937880997d7f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal\" (UID: \"563071327f3cc335e24937880997d7f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.699990 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.699954 6507 apiserver.go:52] "Watching apiserver" Oct 29 23:13:51.779798 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.779769 6507 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 23:13:51.782548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.781804 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0","openshift-network-diagnostics/network-check-target-69jbt","openshift-monitoring/openshift-state-metrics-7bd7489957-j467d","openshift-multus/network-metrics-daemon-xwn99","kube-system/konnectivity-agent-klctw","openshift-cluster-node-tuning-operator/tuned-ccm62","openshift-ingress-canary/ingress-canary-7g2cz","openshift-multus/multus-additional-cni-plugins-fn78d","openshift-ovn-kubernetes/ovnkube-node-xgnjg","kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal","openshift-multus/multus-ltlxg","openshift-network-operator/iptables-alerter-twqrq","kube-system/global-pull-secret-syncer-xcszc","openshift-image-registry/node-ca-tr7bl","openshift-dns/dns-default-5bzwq","openshift-dns/node-resolver-f456x","openshift-monitoring/alertmanager-main-0","openshift-console/console-5cc6cb8466-tphkb","openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal","openshift-monitoring/node-exporter-8qc8d","openshift-insights/insights-runtime-extractor-mmg8f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt"] Oct 29 23:13:51.797180 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.797150 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 23:13:51.797354 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.797224 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qbkzf\"" Oct 29 23:13:51.797468 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.797439 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 23:13:51.797532 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.797471 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-phkn8\"" Oct 29 23:13:51.797532 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.797520 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 23:13:51.797624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.797475 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jb6zf\"" Oct 29 23:13:51.797624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.797547 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 23:13:51.797624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.797556 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 23:13:51.797624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.797550 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 23:13:51.797624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.797552 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 23:13:51.797624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.797593 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 23:13:51.797889 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.797632 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 23:13:51.807693 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.807672 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 23:13:51.808009 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.807989 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 23:13:51.808370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.808346 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7zkzr\"" Oct 29 23:13:51.808530 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.808512 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-t8tv9\"" Oct 29 23:13:51.808726 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.808683 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 23:13:51.808818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.808759 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9w58g\"" Oct 29 23:13:51.808818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.808774 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pnrbh\"" Oct 29 23:13:51.808920 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.808828 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hwcbb\"" Oct 29 23:13:51.808966 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.808920 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-89z4k\"" Oct 29 23:13:51.809001 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.808975 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 23:13:51.809144 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.809126 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8km5l\"" Oct 29 23:13:51.811256 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.811241 6507 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.811335 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.811324 6507 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.811499 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.811487 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 23:13:51.816950 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.816925 6507 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 23:13:51.831097 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.831065 6507 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 23:13:51.831187 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.831119 6507 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-100.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.831676 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.831658 6507 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 23:13:51.831731 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.831697 6507 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-100.ec2.internal" Oct 29 23:13:51.865598 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.865571 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l78l\" (UniqueName: \"kubernetes.io/projected/f4a78746-396b-46fd-a42a-d819d866a6af-kube-api-access-2l78l\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.865778 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.865605 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:51.865778 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.865635 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-var-lib-kubelet\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.865778 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.865657 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-host\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.865778 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.865683 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:51.865778 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.865727 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.865778 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.865740 6507 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 29 23:13:51.866054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.865777 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bcec0d7f-0330-434c-824a-246928597652-multus-daemon-config\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.866054 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.865809 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:52.365789522 +0000 UTC m=+1.728794243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 29 23:13:51.866054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.865834 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-log-socket\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.866054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.865855 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916cb471-df04-4560-aaea-7e9d27f41897-tmp-dir\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:51.866054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.865869 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54576e13-0668-4255-952b-46b2bf5a279c-metrics-certs\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:51.866054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.865881 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-kubernetes\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.866054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.865923 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:51.866054 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.865970 6507 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 29 23:13:51.866054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.865964 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.866054 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.866019 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:52.366005647 +0000 UTC m=+1.729010352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"service-ca" not registered Oct 29 23:13:51.866054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866041 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-cni-bin\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.866054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866059 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-conf-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866062 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916cb471-df04-4560-aaea-7e9d27f41897-tmp-dir\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866082 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-multus-certs\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866117 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-etc-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866145 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-config-out\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866183 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866245 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-config-out\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866285 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866312 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866342 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866346 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866367 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866399 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866477 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.866491 6507 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866494 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-slash\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866524 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm9rx\" (UniqueName: \"kubernetes.io/projected/2c3e5ae7-809c-475e-9623-d4d96e6da95c-kube-api-access-jm9rx\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.866536 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:52.366526049 +0000 UTC m=+1.729530750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-config" not registered Oct 29 23:13:51.866571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866556 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/254e83a6-ab81-4b6b-b5ce-0538ccaede49-dbus\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866581 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85acd560-02f0-4df7-b672-f0c099281e7a-cert\") pod \"ingress-canary-7g2cz\" (UID: \"85acd560-02f0-4df7-b672-f0c099281e7a\") " pod="openshift-ingress-canary/ingress-canary-7g2cz" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866606 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866624 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b21e162-a2f3-4505-9dff-34a47e20d3e6-host-slash\") pod \"iptables-alerter-twqrq\" (UID: \"1b21e162-a2f3-4505-9dff-34a47e20d3e6\") " pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866667 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-tls\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866687 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-system-cni-dir\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866705 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3f3838-8917-45a1-a496-652392caa72c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866728 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866745 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxhf\" (UniqueName: \"kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-kube-api-access-6cxhf\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866760 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-cni-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866787 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-var-lib-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866878 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866903 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85acd560-02f0-4df7-b672-f0c099281e7a-cert\") pod \"ingress-canary-7g2cz\" (UID: \"85acd560-02f0-4df7-b672-f0c099281e7a\") " pod="openshift-ingress-canary/ingress-canary-7g2cz" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.866908 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867007 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867031 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-system-cni-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867050 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73623961-3ffb-4492-a2f4-74a5c4b9551e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867071 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-sys\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867114 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-textfile\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.867166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867133 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxt65\" (UniqueName: \"kubernetes.io/projected/8cccc782-d044-4032-ba8b-8f4d4cdab41b-kube-api-access-cxt65\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867152 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867193 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-textfile\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867204 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6170bf16-e81d-40fd-a4a0-449213b94d19-serviceca\") pod \"node-ca-tr7bl\" (UID: \"6170bf16-e81d-40fd-a4a0-449213b94d19\") " pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867229 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867253 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-etc-kubernetes\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867278 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1b21e162-a2f3-4505-9dff-34a47e20d3e6-iptables-alerter-script\") pod \"iptables-alerter-twqrq\" (UID: \"1b21e162-a2f3-4505-9dff-34a47e20d3e6\") " pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867303 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cnibin\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867329 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcec0d7f-0330-434c-824a-246928597652-cni-binary-copy\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867359 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg42k\" (UniqueName: \"kubernetes.io/projected/bcec0d7f-0330-434c-824a-246928597652-kube-api-access-mg42k\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867392 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvnmz\" (UniqueName: \"kubernetes.io/projected/1b21e162-a2f3-4505-9dff-34a47e20d3e6-kube-api-access-hvnmz\") pod \"iptables-alerter-twqrq\" (UID: \"1b21e162-a2f3-4505-9dff-34a47e20d3e6\") " pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867418 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/73623961-3ffb-4492-a2f4-74a5c4b9551e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867444 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867488 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/916cb471-df04-4560-aaea-7e9d27f41897-metrics-tls\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867513 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867535 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-web-config\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867539 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/73623961-3ffb-4492-a2f4-74a5c4b9551e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867596 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867647 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867676 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-cnibin\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867715 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.867744 6507 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867750 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a00edc7-9624-4236-bb7c-9ac71e911416-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867784 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/916cb471-df04-4560-aaea-7e9d27f41897-metrics-tls\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.867796 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:52.36778294 +0000 UTC m=+1.730787663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-oauth-config" not registered Oct 29 23:13:51.867825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867824 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-lib-modules\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867843 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qspl\" (UniqueName: \"kubernetes.io/projected/85acd560-02f0-4df7-b672-f0c099281e7a-kube-api-access-4qspl\") pod \"ingress-canary-7g2cz\" (UID: \"85acd560-02f0-4df7-b672-f0c099281e7a\") " pod="openshift-ingress-canary/ingress-canary-7g2cz" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867858 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867921 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867995 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868018 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/254e83a6-ab81-4b6b-b5ce-0538ccaede49-kubelet-config\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868036 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc84k\" (UniqueName: \"kubernetes.io/projected/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-kube-api-access-wc84k\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.867997 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config-out\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868071 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868090 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-env-overrides\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868108 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a00edc7-9624-4236-bb7c-9ac71e911416-data-volume\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868127 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-run\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868143 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868159 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868175 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysconfig\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868176 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a00edc7-9624-4236-bb7c-9ac71e911416-data-volume\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868190 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-root\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868293 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-wtmp\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868329 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868356 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-systemd-units\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868385 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868415 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/defafe8e-3565-464e-b2be-522d526248a4-tmp\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868440 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-sys-fs\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868484 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/76e09503-45e8-4e94-9a3b-a5d291a3990f-hosts-file\") pod \"node-resolver-f456x\" (UID: \"76e09503-45e8-4e94-9a3b-a5d291a3990f\") " pod="openshift-dns/node-resolver-f456x" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868499 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-systemd\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868500 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/defafe8e-3565-464e-b2be-522d526248a4-tmp\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868527 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868555 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868571 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868605 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-run-netns\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868683 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-node-log\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868719 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a00edc7-9624-4236-bb7c-9ac71e911416-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868744 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6170bf16-e81d-40fd-a4a0-449213b94d19-host\") pod \"node-ca-tr7bl\" (UID: \"6170bf16-e81d-40fd-a4a0-449213b94d19\") " pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868773 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868813 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-k8s-cni-cncf-io\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868839 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.868862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868868 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34499222-f746-4e87-8dc8-54e6a25c75d6-ovn-node-metrics-cert\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868890 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-modprobe-d\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868915 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-tuned\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868947 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e3f3838-8917-45a1-a496-652392caa72c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.868978 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869000 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-tuned\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869003 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-hostroot\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869056 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-ovnkube-script-lib\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869083 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysctl-d\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869113 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e3f3838-8917-45a1-a496-652392caa72c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869149 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-netns\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869178 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869233 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr5kp\" (UniqueName: \"kubernetes.io/projected/76e09503-45e8-4e94-9a3b-a5d291a3990f-kube-api-access-vr5kp\") pod \"node-resolver-f456x\" (UID: \"76e09503-45e8-4e94-9a3b-a5d291a3990f\") " pod="openshift-dns/node-resolver-f456x" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869254 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-socket-dir-parent\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869273 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-ovnkube-config\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869309 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-ovn\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869338 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869391 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-858x5\" (UniqueName: \"kubernetes.io/projected/916cb471-df04-4560-aaea-7e9d27f41897-kube-api-access-858x5\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869425 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869493 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869511 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnbnk\" (UniqueName: \"kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869529 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869556 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6cba05f-3979-445f-a517-da054562cc20-konnectivity-ca\") pod \"konnectivity-agent-klctw\" (UID: \"e6cba05f-3979-445f-a517-da054562cc20\") " pod="kube-system/konnectivity-agent-klctw" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.869572 6507 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869582 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-cni-netd\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869610 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8zs9\" (UniqueName: \"kubernetes.io/projected/54576e13-0668-4255-952b-46b2bf5a279c-kube-api-access-j8zs9\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.869616 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:52.369605989 +0000 UTC m=+1.732610720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-serving-cert" not registered Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869642 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869717 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869719 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869752 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-cni-multus\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869799 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56n47\" (UniqueName: \"kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47\") pod \"network-check-target-69jbt\" (UID: \"0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc\") " pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869860 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-device-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869880 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e09503-45e8-4e94-9a3b-a5d291a3990f-tmp-dir\") pod \"node-resolver-f456x\" (UID: \"76e09503-45e8-4e94-9a3b-a5d291a3990f\") " pod="openshift-dns/node-resolver-f456x" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869897 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869936 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-systemd\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.869991 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-socket-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870017 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-registration-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870018 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e09503-45e8-4e94-9a3b-a5d291a3990f-tmp-dir\") pod \"node-resolver-f456x\" (UID: \"76e09503-45e8-4e94-9a3b-a5d291a3990f\") " pod="openshift-dns/node-resolver-f456x" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870070 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/916cb471-df04-4560-aaea-7e9d27f41897-config-volume\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870098 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a00edc7-9624-4236-bb7c-9ac71e911416-crio-socket\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870120 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysctl-conf\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.870152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870168 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cccc782-d044-4032-ba8b-8f4d4cdab41b-metrics-client-ca\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870197 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870244 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.870262 6507 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870286 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.870335 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:52.370321166 +0000 UTC m=+1.733325876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"oauth-serving-cert" not registered Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870367 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6cba05f-3979-445f-a517-da054562cc20-agent-certs\") pod \"konnectivity-agent-klctw\" (UID: \"e6cba05f-3979-445f-a517-da054562cc20\") " pod="kube-system/konnectivity-agent-klctw" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870398 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870475 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870499 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870518 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-kubelet\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870536 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np8k4\" (UniqueName: \"kubernetes.io/projected/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-api-access-np8k4\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870628 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv2xt\" (UniqueName: \"kubernetes.io/projected/6170bf16-e81d-40fd-a4a0-449213b94d19-kube-api-access-zv2xt\") pod \"node-ca-tr7bl\" (UID: \"6170bf16-e81d-40fd-a4a0-449213b94d19\") " pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870747 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-kubelet\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870783 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-cni-bin\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870808 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-sys\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870858 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c3e5ae7-809c-475e-9623-d4d96e6da95c-tmp\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870898 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870933 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c3e5ae7-809c-475e-9623-d4d96e6da95c-tmp\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.870961 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbnn\" (UniqueName: \"kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-kube-api-access-tnbnn\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.871063 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k4q5\" (UniqueName: \"kubernetes.io/projected/7a00edc7-9624-4236-bb7c-9ac71e911416-kube-api-access-9k4q5\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.871184 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6flh\" (UniqueName: \"kubernetes.io/projected/7e3f3838-8917-45a1-a496-652392caa72c-kube-api-access-v6flh\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:51.871555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.871214 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kw4s\" (UniqueName: \"kubernetes.io/projected/34499222-f746-4e87-8dc8-54e6a25c75d6-kube-api-access-4kw4s\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.894761 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.894737 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l78l\" (UniqueName: \"kubernetes.io/projected/f4a78746-396b-46fd-a42a-d819d866a6af-kube-api-access-2l78l\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.894895 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.894879 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg42k\" (UniqueName: \"kubernetes.io/projected/bcec0d7f-0330-434c-824a-246928597652-kube-api-access-mg42k\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.896722 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.896707 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-858x5\" (UniqueName: \"kubernetes.io/projected/916cb471-df04-4560-aaea-7e9d27f41897-kube-api-access-858x5\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:51.896789 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.896713 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56n47\" (UniqueName: \"kubernetes.io/projected/0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc-kube-api-access-56n47\") pod \"network-check-target-69jbt\" (UID: \"0e9b5b65-2a88-4b83-9cf4-d8f09aba19bc\") " pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:13:51.898257 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.898237 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8zs9\" (UniqueName: \"kubernetes.io/projected/54576e13-0668-4255-952b-46b2bf5a279c-kube-api-access-j8zs9\") pod \"network-metrics-daemon-xwn99\" (UID: \"54576e13-0668-4255-952b-46b2bf5a279c\") " pod="openshift-multus/network-metrics-daemon-xwn99" Oct 29 23:13:51.898332 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.898257 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr5kp\" (UniqueName: \"kubernetes.io/projected/76e09503-45e8-4e94-9a3b-a5d291a3990f-kube-api-access-vr5kp\") pod \"node-resolver-f456x\" (UID: \"76e09503-45e8-4e94-9a3b-a5d291a3990f\") " pod="openshift-dns/node-resolver-f456x" Oct 29 23:13:51.898723 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.898706 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 23:13:51.898796 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.898778 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 23:13:51.898883 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.898839 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 23:13:51.899090 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899058 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 23:13:51.899090 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899078 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 29 23:13:51.899090 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899084 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 23:13:51.899528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899394 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 23:13:51.899528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899395 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 29 23:13:51.899528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899480 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 23:13:51.899528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899451 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 29 23:13:51.899528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899518 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 23:13:51.899739 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899618 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 23:13:51.899739 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899623 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 23:13:51.899739 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899725 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 23:13:51.899739 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899737 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 23:13:51.899914 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899740 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 23:13:51.899914 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899797 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 23:13:51.899914 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899832 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 23:13:51.899914 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899828 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6cba05f-3979-445f-a517-da054562cc20-konnectivity-ca\") pod \"konnectivity-agent-klctw\" (UID: \"e6cba05f-3979-445f-a517-da054562cc20\") " pod="kube-system/konnectivity-agent-klctw" Oct 29 23:13:51.899914 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899881 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 23:13:51.900101 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899921 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.900101 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.899938 6507 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 29 23:13:51.900101 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.899983 6507 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 29 23:13:51.900101 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.899998 6507 projected.go:196] Error preparing data for projected volume kube-api-access-jnbnk for pod openshift-console/console-5cc6cb8466-tphkb: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 29 23:13:51.900101 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.899983 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 23:13:51.900253 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:51.900121 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:52.400064951 +0000 UTC m=+1.763069675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jnbnk" (UniqueName: "kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 29 23:13:51.900253 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.900127 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 29 23:13:51.900253 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.900213 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 29 23:13:51.900386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.900309 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 29 23:13:51.900386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.900313 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 29 23:13:51.900386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.900335 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 29 23:13:51.900386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.900339 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc84k\" (UniqueName: \"kubernetes.io/projected/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-kube-api-access-wc84k\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.900386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.900342 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/916cb471-df04-4560-aaea-7e9d27f41897-config-volume\") pod \"dns-default-5bzwq\" (UID: \"916cb471-df04-4560-aaea-7e9d27f41897\") " pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:51.900386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.900339 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 29 23:13:51.900743 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.900731 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 29 23:13:51.900944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.900928 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 29 23:13:51.903943 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.903928 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2vcdm\"" Oct 29 23:13:51.904007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.903957 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 23:13:51.904060 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.904006 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 23:13:51.904060 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.904015 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 23:13:51.905648 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.905633 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tldxj\"" Oct 29 23:13:51.905738 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.905725 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 23:13:51.906082 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.906065 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm9rx\" (UniqueName: \"kubernetes.io/projected/2c3e5ae7-809c-475e-9623-d4d96e6da95c-kube-api-access-jm9rx\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.906126 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.906066 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv2xt\" (UniqueName: \"kubernetes.io/projected/6170bf16-e81d-40fd-a4a0-449213b94d19-kube-api-access-zv2xt\") pod \"node-ca-tr7bl\" (UID: \"6170bf16-e81d-40fd-a4a0-449213b94d19\") " pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:13:51.906188 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.906134 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qspl\" (UniqueName: \"kubernetes.io/projected/85acd560-02f0-4df7-b672-f0c099281e7a-kube-api-access-4qspl\") pod \"ingress-canary-7g2cz\" (UID: \"85acd560-02f0-4df7-b672-f0c099281e7a\") " pod="openshift-ingress-canary/ingress-canary-7g2cz" Oct 29 23:13:51.906242 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.906220 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bcec0d7f-0330-434c-824a-246928597652-multus-daemon-config\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.906291 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.906272 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cni-binary-copy\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.906916 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.906900 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.907512 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.907494 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 29 23:13:51.907635 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.907617 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.907696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.907640 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 29 23:13:51.907696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.907621 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 23:13:51.907696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.907645 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcec0d7f-0330-434c-824a-246928597652-cni-binary-copy\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.907696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.907647 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1b21e162-a2f3-4505-9dff-34a47e20d3e6-iptables-alerter-script\") pod \"iptables-alerter-twqrq\" (UID: \"1b21e162-a2f3-4505-9dff-34a47e20d3e6\") " pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:13:51.907696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.907690 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 29 23:13:51.907921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.907725 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 29 23:13:51.907921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.907766 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 29 23:13:51.907921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.907775 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 29 23:13:51.907921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.907804 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 29 23:13:51.907921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.907824 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 29 23:13:51.908153 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.907954 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6170bf16-e81d-40fd-a4a0-449213b94d19-serviceca\") pod \"node-ca-tr7bl\" (UID: \"6170bf16-e81d-40fd-a4a0-449213b94d19\") " pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:13:51.908153 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908120 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ejptm05c9424c\"" Oct 29 23:13:51.908153 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908100 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a00edc7-9624-4236-bb7c-9ac71e911416-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:51.908153 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908126 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-web-config\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.908331 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908232 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.908331 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908275 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-env-overrides\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.908331 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908304 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np8k4\" (UniqueName: \"kubernetes.io/projected/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-api-access-np8k4\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:51.908416 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.907802 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:51.908416 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908386 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.908870 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908774 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 29 23:13:51.908870 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908786 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.908870 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908785 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnbnn\" (UniqueName: \"kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-kube-api-access-tnbnn\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.908870 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908818 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 29 23:13:51.908870 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908829 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 29 23:13:51.908870 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908845 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 29 23:13:51.908870 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908848 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 29 23:13:51.909134 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908893 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 29 23:13:51.909134 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908900 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.909134 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.908919 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.909134 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.909128 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kw4s\" (UniqueName: \"kubernetes.io/projected/34499222-f746-4e87-8dc8-54e6a25c75d6-kube-api-access-4kw4s\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.909312 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.909292 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e3f3838-8917-45a1-a496-652392caa72c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:51.909348 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.909295 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.909502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.909488 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k4q5\" (UniqueName: \"kubernetes.io/projected/7a00edc7-9624-4236-bb7c-9ac71e911416-kube-api-access-9k4q5\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:51.909638 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.909625 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-ovnkube-config\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.909730 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.909716 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.910153 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.910138 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 29 23:13:51.910443 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.910338 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 23:13:51.910443 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.910355 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 23:13:51.910443 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.910395 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2vtzc\"" Oct 29 23:13:51.910443 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.910395 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 23:13:51.910443 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.910414 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sx9m6\"" Oct 29 23:13:51.910721 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.910487 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.910721 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.910508 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gj5kk\"" Oct 29 23:13:51.910721 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.910671 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.910863 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.910752 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.910863 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.910755 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.910863 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.910782 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6flh\" (UniqueName: \"kubernetes.io/projected/7e3f3838-8917-45a1-a496-652392caa72c-kube-api-access-v6flh\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:51.910964 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.910942 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vvv78\"" Oct 29 23:13:51.911017 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.910981 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rkfgr\"" Oct 29 23:13:51.911017 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.911002 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vwjx7\"" Oct 29 23:13:51.911107 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.911019 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 29 23:13:51.911107 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.911056 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 29 23:13:51.911107 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.911058 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-59hcs\"" Oct 29 23:13:51.911186 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.911168 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.911294 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.911264 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 23:13:51.911294 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.911277 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 23:13:51.911294 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.911275 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pgvzg\"" Oct 29 23:13:51.911413 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.911278 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 23:13:51.911413 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.911271 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 29 23:13:51.911413 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.911291 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 29 23:13:51.915483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.915449 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvnmz\" (UniqueName: \"kubernetes.io/projected/1b21e162-a2f3-4505-9dff-34a47e20d3e6-kube-api-access-hvnmz\") pod \"iptables-alerter-twqrq\" (UID: \"1b21e162-a2f3-4505-9dff-34a47e20d3e6\") " pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:13:51.915807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.915792 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxt65\" (UniqueName: \"kubernetes.io/projected/8cccc782-d044-4032-ba8b-8f4d4cdab41b-kube-api-access-cxt65\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.915932 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.915919 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.917484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.917196 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3f3838-8917-45a1-a496-652392caa72c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:51.917484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.917213 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-tls\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.917484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.917248 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.917484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.917249 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:51.917484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.917394 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73623961-3ffb-4492-a2f4-74a5c4b9551e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:51.917484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.917419 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.917484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.917421 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.917864 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.917491 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.917864 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.917790 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxhf\" (UniqueName: \"kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-kube-api-access-6cxhf\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.917982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.917962 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.918035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.917978 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.918611 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.918594 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.918611 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.918600 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-web-config\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.918693 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.918604 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.919225 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.919210 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a00edc7-9624-4236-bb7c-9ac71e911416-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:51.919278 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.919222 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34499222-f746-4e87-8dc8-54e6a25c75d6-ovn-node-metrics-cert\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.919341 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.919324 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34499222-f746-4e87-8dc8-54e6a25c75d6-ovnkube-script-lib\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.919385 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.919364 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/73623961-3ffb-4492-a2f4-74a5c4b9551e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qs5th\" (UID: \"73623961-3ffb-4492-a2f4-74a5c4b9551e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qs5th" Oct 29 23:13:51.919493 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.919453 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e3f3838-8917-45a1-a496-652392caa72c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-j467d\" (UID: \"7e3f3838-8917-45a1-a496-652392caa72c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-j467d" Oct 29 23:13:51.922837 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.919988 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/254e83a6-ab81-4b6b-b5ce-0538ccaede49-original-pull-secret\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:51.922837 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.920860 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6cba05f-3979-445f-a517-da054562cc20-agent-certs\") pod \"konnectivity-agent-klctw\" (UID: \"e6cba05f-3979-445f-a517-da054562cc20\") " pod="kube-system/konnectivity-agent-klctw" Oct 29 23:13:51.922837 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.921043 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cccc782-d044-4032-ba8b-8f4d4cdab41b-metrics-client-ca\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.923301 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.923276 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 29 23:13:51.923389 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.923312 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 29 23:13:51.923443 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.923429 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 29 23:13:51.929320 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.929300 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.929918 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.929900 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:51.930440 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.930424 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:13:51.972280 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972208 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-hostroot\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.972280 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972241 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysctl-d\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.972280 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972260 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-netns\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.972280 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972277 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-socket-dir-parent\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972297 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-ovn\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972312 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-hostroot\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972345 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-ovn\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972347 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-netns\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972362 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-socket-dir-parent\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972378 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972414 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysctl-d\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972471 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-cni-netd\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972483 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972503 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-cni-multus\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972522 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-device-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972547 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-systemd\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972509 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-cni-netd\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972567 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-device-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972570 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-socket-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972565 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-cni-multus\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.972591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972593 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-systemd\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972620 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-socket-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972624 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-registration-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972659 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a00edc7-9624-4236-bb7c-9ac71e911416-crio-socket\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972707 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysctl-conf\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972709 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-registration-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972746 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a00edc7-9624-4236-bb7c-9ac71e911416-crio-socket\") pod \"insights-runtime-extractor-mmg8f\" (UID: \"7a00edc7-9624-4236-bb7c-9ac71e911416\") " pod="openshift-insights/insights-runtime-extractor-mmg8f" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972801 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972840 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972867 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-kubelet\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972883 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-kubelet\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972889 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysctl-conf\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972897 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-cni-bin\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972928 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-kubelet\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972931 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-cni-bin\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972943 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-kubelet\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972952 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-sys\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.972983 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-var-lib-kubelet\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973005 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-host\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973017 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-sys\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973036 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-var-lib-kubelet\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973041 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-log-socket\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973089 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-host\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973091 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-kubernetes\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973114 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-log-socket\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973121 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-kubernetes\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973155 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-cni-bin\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973175 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-conf-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973193 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-multus-certs\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973209 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-etc-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973235 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973247 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-var-lib-cni-bin\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973261 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-conf-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973267 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-slash\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973289 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-slash\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973296 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/254e83a6-ab81-4b6b-b5ce-0538ccaede49-dbus\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973306 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973313 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-etc-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973323 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b21e162-a2f3-4505-9dff-34a47e20d3e6-host-slash\") pod \"iptables-alerter-twqrq\" (UID: \"1b21e162-a2f3-4505-9dff-34a47e20d3e6\") " pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973338 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-multus-certs\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973358 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/254e83a6-ab81-4b6b-b5ce-0538ccaede49-dbus\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973364 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-system-cni-dir\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973377 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b21e162-a2f3-4505-9dff-34a47e20d3e6-host-slash\") pod \"iptables-alerter-twqrq\" (UID: \"1b21e162-a2f3-4505-9dff-34a47e20d3e6\") " pod="openshift-network-operator/iptables-alerter-twqrq" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973406 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-system-cni-dir\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973434 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-cni-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973449 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-var-lib-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973492 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-system-cni-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973506 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-multus-cni-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973511 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-sys\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973527 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-var-lib-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973541 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-sys\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973548 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-system-cni-dir\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973552 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-etc-kubernetes\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973571 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-etc-kubernetes\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973578 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cnibin\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973613 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-cnibin\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973636 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.973649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973644 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9-cnibin\") pod \"multus-additional-cni-plugins-fn78d\" (UID: \"85fb3a66-9c1e-4acc-bd3f-82b115c2b4f9\") " pod="openshift-multus/multus-additional-cni-plugins-fn78d" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973672 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-run-openvswitch\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973702 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-cnibin\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973721 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-lib-modules\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973760 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/254e83a6-ab81-4b6b-b5ce-0538ccaede49-kubelet-config\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973780 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-run\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973813 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-lib-modules\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973819 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysconfig\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973851 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-root\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973859 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-run\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973861 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/254e83a6-ab81-4b6b-b5ce-0538ccaede49-kubelet-config\") pod \"global-pull-secret-syncer-xcszc\" (UID: \"254e83a6-ab81-4b6b-b5ce-0538ccaede49\") " pod="kube-system/global-pull-secret-syncer-xcszc" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973860 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-sysconfig\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973878 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-wtmp\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973889 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-root\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973911 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8cccc782-d044-4032-ba8b-8f4d4cdab41b-node-exporter-wtmp\") pod \"node-exporter-8qc8d\" (UID: \"8cccc782-d044-4032-ba8b-8f4d4cdab41b\") " pod="openshift-monitoring/node-exporter-8qc8d" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973911 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-systemd-units\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973928 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-systemd-units\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973937 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-sys-fs\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973957 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/76e09503-45e8-4e94-9a3b-a5d291a3990f-hosts-file\") pod \"node-resolver-f456x\" (UID: \"76e09503-45e8-4e94-9a3b-a5d291a3990f\") " pod="openshift-dns/node-resolver-f456x" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973987 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-systemd\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.973997 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/76e09503-45e8-4e94-9a3b-a5d291a3990f-hosts-file\") pod \"node-resolver-f456x\" (UID: \"76e09503-45e8-4e94-9a3b-a5d291a3990f\") " pod="openshift-dns/node-resolver-f456x" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.974001 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4a78746-396b-46fd-a42a-d819d866a6af-sys-fs\") pod \"aws-ebs-csi-driver-node-hlgrt\" (UID: \"f4a78746-396b-46fd-a42a-d819d866a6af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlgrt" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.974027 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-run-netns\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.974052 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-run-netns\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.974077 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-systemd\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.974101 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-node-log\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.974134 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6170bf16-e81d-40fd-a4a0-449213b94d19-host\") pod \"node-ca-tr7bl\" (UID: \"6170bf16-e81d-40fd-a4a0-449213b94d19\") " pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.974172 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-node-log\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.974175 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-k8s-cni-cncf-io\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.974204 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.974208 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bcec0d7f-0330-434c-824a-246928597652-host-run-k8s-cni-cncf-io\") pod \"multus-ltlxg\" (UID: \"bcec0d7f-0330-434c-824a-246928597652\") " pod="openshift-multus/multus-ltlxg" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.974176 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6170bf16-e81d-40fd-a4a0-449213b94d19-host\") pod \"node-ca-tr7bl\" (UID: \"6170bf16-e81d-40fd-a4a0-449213b94d19\") " pod="openshift-image-registry/node-ca-tr7bl" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.974233 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-modprobe-d\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.974271 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.974280 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.974300 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2c3e5ae7-809c-475e-9623-d4d96e6da95c-etc-modprobe-d\") pod \"tuned-ccm62\" (UID: \"2c3e5ae7-809c-475e-9623-d4d96e6da95c\") " pod="openshift-cluster-node-tuning-operator/tuned-ccm62" Oct 29 23:13:51.974680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:51.974347 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34499222-f746-4e87-8dc8-54e6a25c75d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-xgnjg\" (UID: \"34499222-f746-4e87-8dc8-54e6a25c75d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:52.377770 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:52.377688 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:52.377770 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:52.377727 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:52.377770 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:52.377758 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:52.377998 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:52.377799 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:52.377998 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:52.377811 6507 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 29 23:13:52.377998 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:52.377830 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:52.377998 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:52.377813 6507 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 29 23:13:52.377998 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:52.377870 6507 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 29 23:13:52.377998 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:52.377882 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:52.377998 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:52.377907 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:53.377894607 +0000 UTC m=+2.740899307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-config" not registered Oct 29 23:13:52.377998 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:52.377924 6507 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 29 23:13:52.377998 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:52.377925 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:53.377917944 +0000 UTC m=+2.740922644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"service-ca" not registered Oct 29 23:13:52.377998 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:52.377952 6507 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 29 23:13:52.377998 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:52.377967 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:53.377955227 +0000 UTC m=+2.740959932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-oauth-config" not registered Oct 29 23:13:52.377998 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:52.377981 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:53.377975647 +0000 UTC m=+2.740980348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 29 23:13:52.377998 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:52.378006 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:53.377990976 +0000 UTC m=+2.740995692 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-serving-cert" not registered Oct 29 23:13:52.378370 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:52.378026 6507 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 29 23:13:52.378370 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:52.378053 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:53.378047192 +0000 UTC m=+2.741051892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"oauth-serving-cert" not registered Oct 29 23:13:52.471943 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:52.471911 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5bzwq" Oct 29 23:13:52.479369 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:52.479349 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnbnk\" (UniqueName: \"kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:52.479499 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:52.479485 6507 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 29 23:13:52.479535 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:52.479502 6507 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 29 23:13:52.479535 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:52.479511 6507 projected.go:196] Error preparing data for projected volume kube-api-access-jnbnk for pod openshift-console/console-5cc6cb8466-tphkb: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 29 23:13:52.479626 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:52.479555 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:53.479542732 +0000 UTC m=+2.842547433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jnbnk" (UniqueName: "kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 29 23:13:53.390647 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:53.390604 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:53.391107 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:53.390664 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:53.391107 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:53.390710 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:53.391107 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:53.390732 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:53.391107 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:53.390744 6507 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 29 23:13:53.391107 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:53.390770 6507 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 29 23:13:53.391107 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:53.390809 6507 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 29 23:13:53.391107 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:53.390754 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:53.391107 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:53.390822 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:55.390805737 +0000 UTC m=+4.753810460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"oauth-serving-cert" not registered Oct 29 23:13:53.391107 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:53.390834 6507 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 29 23:13:53.391107 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:53.390868 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:55.390841458 +0000 UTC m=+4.753846170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-serving-cert" not registered Oct 29 23:13:53.391107 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:53.390865 6507 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 29 23:13:53.391107 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:53.390886 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:55.390879226 +0000 UTC m=+4.753883927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 29 23:13:53.391107 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:53.390900 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:55.39089296 +0000 UTC m=+4.753897662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-config" not registered Oct 29 23:13:53.391107 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:53.390953 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:55.390934687 +0000 UTC m=+4.753939404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"service-ca" not registered Oct 29 23:13:53.391107 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:53.390979 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:53.391107 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:53.391113 6507 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 29 23:13:53.391686 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:53.391158 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:55.391146905 +0000 UTC m=+4.754151609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-oauth-config" not registered Oct 29 23:13:53.491701 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:53.491671 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnbnk\" (UniqueName: \"kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:53.491840 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:53.491799 6507 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 29 23:13:53.491840 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:53.491815 6507 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 29 23:13:53.491840 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:53.491825 6507 projected.go:196] Error preparing data for projected volume kube-api-access-jnbnk for pod openshift-console/console-5cc6cb8466-tphkb: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 29 23:13:53.491947 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:53.491870 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:55.491857935 +0000 UTC m=+4.854862649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jnbnk" (UniqueName: "kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 29 23:13:54.854959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:54.854930 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:55.413341 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:55.413296 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:55.413561 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:55.413359 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:55.413561 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:55.413416 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:55.413561 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:55.413416 6507 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 29 23:13:55.413561 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:55.413484 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:55.413561 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:55.413508 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:59.413490781 +0000 UTC m=+8.776495492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-config" not registered Oct 29 23:13:55.413561 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:55.413534 6507 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 29 23:13:55.413561 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:55.413537 6507 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 29 23:13:55.413561 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:55.413552 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:55.413906 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:55.413573 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:55.413906 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:55.413536 6507 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 29 23:13:55.413906 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:55.413586 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:59.413572378 +0000 UTC m=+8.776577082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-serving-cert" not registered Oct 29 23:13:55.413906 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:55.413605 6507 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 29 23:13:55.413906 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:55.413618 6507 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 29 23:13:55.413906 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:55.413607 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:59.413600975 +0000 UTC m=+8.776605676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"oauth-serving-cert" not registered Oct 29 23:13:55.413906 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:55.413669 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:59.413646813 +0000 UTC m=+8.776651525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-oauth-config" not registered Oct 29 23:13:55.413906 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:55.413690 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:59.413680387 +0000 UTC m=+8.776685090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 29 23:13:55.413906 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:55.413710 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:59.413698799 +0000 UTC m=+8.776703502 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"service-ca" not registered Oct 29 23:13:55.514863 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:55.514822 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnbnk\" (UniqueName: \"kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:55.515028 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:55.514955 6507 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 29 23:13:55.515028 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:55.514970 6507 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 29 23:13:55.515028 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:55.514978 6507 projected.go:196] Error preparing data for projected volume kube-api-access-jnbnk for pod openshift-console/console-5cc6cb8466-tphkb: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 29 23:13:55.515133 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:55.515032 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:59.515020497 +0000 UTC m=+8.878025211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jnbnk" (UniqueName: "kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 29 23:13:56.994246 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:56.994208 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:57.009354 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:57.009328 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:13:58.488840 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:58.488808 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:13:58.489209 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:58.489050 6507 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 23:13:59.452098 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:59.452041 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:59.452302 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:59.452118 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:59.452302 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:59.452153 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:59.452302 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:59.452176 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:59.452302 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:59.452196 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:59.452302 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:59.452203 6507 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 29 23:13:59.452302 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:59.452245 6507 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 29 23:13:59.452302 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:59.452251 6507 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 29 23:13:59.452302 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:59.452257 6507 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 29 23:13:59.452302 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:59.452290 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:07.452268059 +0000 UTC m=+16.815272771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-oauth-config" not registered Oct 29 23:13:59.452302 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:59.452309 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:07.452302173 +0000 UTC m=+16.815306874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 29 23:13:59.452872 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:59.452321 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:07.452316223 +0000 UTC m=+16.815320923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"oauth-serving-cert" not registered Oct 29 23:13:59.452872 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:59.452328 6507 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 29 23:13:59.452872 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:59.452338 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:59.452872 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:59.452357 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:07.452344806 +0000 UTC m=+16.815349548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"service-ca" not registered Oct 29 23:13:59.452872 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:59.452360 6507 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 29 23:13:59.452872 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:59.452395 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:07.452387306 +0000 UTC m=+16.815392006 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-serving-cert" not registered Oct 29 23:13:59.452872 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:59.452408 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:07.452402553 +0000 UTC m=+16.815407254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-config" not registered Oct 29 23:13:59.553553 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:59.553510 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnbnk\" (UniqueName: \"kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:13:59.553935 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:59.553673 6507 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 29 23:13:59.553935 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:59.553694 6507 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 29 23:13:59.553935 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:59.553704 6507 projected.go:196] Error preparing data for projected volume kube-api-access-jnbnk for pod openshift-console/console-5cc6cb8466-tphkb: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 29 23:13:59.553935 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:13:59.553754 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:07.553741607 +0000 UTC m=+16.916746332 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jnbnk" (UniqueName: "kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 29 23:13:59.757642 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:13:59.757559 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:14:01.051711 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:01.051629 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xgnjg" Oct 29 23:14:07.525806 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:07.525762 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:14:07.526294 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:07.525822 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:14:07.526294 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:07.525873 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:14:07.526294 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:07.525902 6507 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 29 23:14:07.526294 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:07.525957 6507 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 29 23:14:07.526294 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:07.525986 6507 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 29 23:14:07.526294 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:07.525996 6507 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 29 23:14:07.526294 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:07.525909 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:14:07.526294 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:07.525991 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:23.525967284 +0000 UTC m=+32.888972010 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-config" not registered Oct 29 23:14:07.526294 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:07.526099 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:23.526084586 +0000 UTC m=+32.889089287 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"oauth-serving-cert" not registered Oct 29 23:14:07.526294 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:07.526124 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:23.526114604 +0000 UTC m=+32.889119308 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-serving-cert" not registered Oct 29 23:14:07.526294 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:07.526144 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:23.526136258 +0000 UTC m=+32.889140961 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"console-oauth-config" not registered Oct 29 23:14:07.526294 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:07.526199 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:14:07.526294 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:07.526230 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:14:07.526294 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:07.526304 6507 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 29 23:14:07.526761 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:07.526307 6507 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 29 23:14:07.526761 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:07.526345 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:23.526334209 +0000 UTC m=+32.889338928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"service-ca" not registered Oct 29 23:14:07.526761 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:07.526363 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:23.526351192 +0000 UTC m=+32.889355893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 29 23:14:07.627239 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:07.627202 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnbnk\" (UniqueName: \"kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk\") pod \"console-5cc6cb8466-tphkb\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:14:07.627389 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:07.627328 6507 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 29 23:14:07.627389 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:07.627345 6507 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 29 23:14:07.627389 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:07.627354 6507 projected.go:196] Error preparing data for projected volume kube-api-access-jnbnk for pod openshift-console/console-5cc6cb8466-tphkb: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 29 23:14:07.627510 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:07.627402 6507 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk podName:defafe8e-3565-464e-b2be-522d526248a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:14:23.627390171 +0000 UTC m=+32.990394885 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jnbnk" (UniqueName: "kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk") pod "console-5cc6cb8466-tphkb" (UID: "defafe8e-3565-464e-b2be-522d526248a4") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 29 23:14:10.254740 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.254705 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 23:14:10.255142 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.255003 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="alertmanager" containerID="cri-o://59250c65c1ead7102d5de10cea540bcec94919334a6eccb71632a3760316cdcf" gracePeriod=120 Oct 29 23:14:10.255142 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.255071 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="kube-rbac-proxy-metric" containerID="cri-o://e01f48c65de1075865455a098a5d1241260e5af5c6f4df20a79ffd89e9be4b5e" gracePeriod=120 Oct 29 23:14:10.255259 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.255114 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="prom-label-proxy" containerID="cri-o://1e5212ad899b6158f44ac73409016545410d98f0b5c9504631292948e52e86d9" gracePeriod=120 Oct 29 23:14:10.255259 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.255148 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="kube-rbac-proxy" containerID="cri-o://372657f190070a2029a6036dd7ad3e45a76fddc8ed6e9fdad04fecacd5b50cda" gracePeriod=120 Oct 29 23:14:10.255259 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.255185 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="config-reloader" containerID="cri-o://aea55638adf89557b3d2f1a784c52c2cda19c440a6f930fa5d3d7bb6d8cfca8a" gracePeriod=120 Oct 29 23:14:10.255575 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.255126 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="kube-rbac-proxy-web" containerID="cri-o://147c61fdc70a55c617acd2b28d32e2e3b32e694f38e2f2fd9ea28692cffce801" gracePeriod=120 Oct 29 23:14:10.881548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.881511 6507 generic.go:356] "Generic (PLEG): container finished" podID="6e2d17bd-2440-419f-9403-73f10f421359" containerID="1e5212ad899b6158f44ac73409016545410d98f0b5c9504631292948e52e86d9" exitCode=0 Oct 29 23:14:10.881548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.881537 6507 generic.go:356] "Generic (PLEG): container finished" podID="6e2d17bd-2440-419f-9403-73f10f421359" containerID="e01f48c65de1075865455a098a5d1241260e5af5c6f4df20a79ffd89e9be4b5e" exitCode=0 Oct 29 23:14:10.881548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.881548 6507 generic.go:356] "Generic (PLEG): container finished" podID="6e2d17bd-2440-419f-9403-73f10f421359" containerID="372657f190070a2029a6036dd7ad3e45a76fddc8ed6e9fdad04fecacd5b50cda" exitCode=0 Oct 29 23:14:10.881548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.881557 6507 generic.go:356] "Generic (PLEG): container finished" podID="6e2d17bd-2440-419f-9403-73f10f421359" containerID="aea55638adf89557b3d2f1a784c52c2cda19c440a6f930fa5d3d7bb6d8cfca8a" exitCode=0 Oct 29 23:14:10.881862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.881565 6507 generic.go:356] "Generic (PLEG): container finished" podID="6e2d17bd-2440-419f-9403-73f10f421359" containerID="59250c65c1ead7102d5de10cea540bcec94919334a6eccb71632a3760316cdcf" exitCode=0 Oct 29 23:14:10.881862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.881577 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e2d17bd-2440-419f-9403-73f10f421359","Type":"ContainerDied","Data":"1e5212ad899b6158f44ac73409016545410d98f0b5c9504631292948e52e86d9"} Oct 29 23:14:10.881862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.881654 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e2d17bd-2440-419f-9403-73f10f421359","Type":"ContainerDied","Data":"e01f48c65de1075865455a098a5d1241260e5af5c6f4df20a79ffd89e9be4b5e"} Oct 29 23:14:10.881862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.881686 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e2d17bd-2440-419f-9403-73f10f421359","Type":"ContainerDied","Data":"372657f190070a2029a6036dd7ad3e45a76fddc8ed6e9fdad04fecacd5b50cda"} Oct 29 23:14:10.881862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.881716 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e2d17bd-2440-419f-9403-73f10f421359","Type":"ContainerDied","Data":"aea55638adf89557b3d2f1a784c52c2cda19c440a6f930fa5d3d7bb6d8cfca8a"} Oct 29 23:14:10.881862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:10.881756 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e2d17bd-2440-419f-9403-73f10f421359","Type":"ContainerDied","Data":"59250c65c1ead7102d5de10cea540bcec94919334a6eccb71632a3760316cdcf"} Oct 29 23:14:11.533705 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.533674 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:14:11.564606 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.564569 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6e2d17bd-2440-419f-9403-73f10f421359\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " Oct 29 23:14:11.564741 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.564621 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-config-volume\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-config-volume\") pod \"6e2d17bd-2440-419f-9403-73f10f421359\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " Oct 29 23:14:11.564741 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.564654 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-cluster-tls-config\") pod \"6e2d17bd-2440-419f-9403-73f10f421359\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " Oct 29 23:14:11.564741 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.564730 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-config-out\") pod \"6e2d17bd-2440-419f-9403-73f10f421359\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " Oct 29 23:14:11.564908 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.564752 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-kube-api-access-tnbnn\" (UniqueName: \"kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-kube-api-access-tnbnn\") pod \"6e2d17bd-2440-419f-9403-73f10f421359\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " Oct 29 23:14:11.564908 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.564784 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-web-config\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-web-config\") pod \"6e2d17bd-2440-419f-9403-73f10f421359\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " Oct 29 23:14:11.564908 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.564833 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-metrics-client-ca\") pod \"6e2d17bd-2440-419f-9403-73f10f421359\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " Oct 29 23:14:11.564908 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.564868 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy\") pod \"6e2d17bd-2440-419f-9403-73f10f421359\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " Oct 29 23:14:11.565651 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.564934 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-web\") pod \"6e2d17bd-2440-419f-9403-73f10f421359\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " Oct 29 23:14:11.565651 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.565037 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-main-db\") pod \"6e2d17bd-2440-419f-9403-73f10f421359\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " Oct 29 23:14:11.565651 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.565077 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-trusted-ca-bundle\") pod \"6e2d17bd-2440-419f-9403-73f10f421359\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " Oct 29 23:14:11.565651 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.565135 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-tls-assets\" (UniqueName: \"kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-tls-assets\") pod \"6e2d17bd-2440-419f-9403-73f10f421359\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " Oct 29 23:14:11.565651 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.565169 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-main-tls\") pod \"6e2d17bd-2440-419f-9403-73f10f421359\" (UID: \"6e2d17bd-2440-419f-9403-73f10f421359\") " Oct 29 23:14:11.565911 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.565662 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-metrics-client-ca" pod "6e2d17bd-2440-419f-9403-73f10f421359" (UID: "6e2d17bd-2440-419f-9403-73f10f421359"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:11.565911 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.565678 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-main-db" pod "6e2d17bd-2440-419f-9403-73f10f421359" (UID: "6e2d17bd-2440-419f-9403-73f10f421359"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:11.566482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.566193 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-trusted-ca-bundle" pod "6e2d17bd-2440-419f-9403-73f10f421359" (UID: "6e2d17bd-2440-419f-9403-73f10f421359"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:11.568593 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.568554 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-config-out" pod "6e2d17bd-2440-419f-9403-73f10f421359" (UID: "6e2d17bd-2440-419f-9403-73f10f421359"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:11.568999 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.568955 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-metric" pod "6e2d17bd-2440-419f-9403-73f10f421359" (UID: "6e2d17bd-2440-419f-9403-73f10f421359"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:11.569095 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.569062 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy" pod "6e2d17bd-2440-419f-9403-73f10f421359" (UID: "6e2d17bd-2440-419f-9403-73f10f421359"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:11.569255 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.569233 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-tls-assets" pod "6e2d17bd-2440-419f-9403-73f10f421359" (UID: "6e2d17bd-2440-419f-9403-73f10f421359"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:14:11.569531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.569504 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-main-tls" pod "6e2d17bd-2440-419f-9403-73f10f421359" (UID: "6e2d17bd-2440-419f-9403-73f10f421359"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:11.569659 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.569627 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-config-volume" pod "6e2d17bd-2440-419f-9403-73f10f421359" (UID: "6e2d17bd-2440-419f-9403-73f10f421359"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:11.569744 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.569719 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-web" pod "6e2d17bd-2440-419f-9403-73f10f421359" (UID: "6e2d17bd-2440-419f-9403-73f10f421359"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:11.569850 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.569833 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-kube-api-access-tnbnn" pod "6e2d17bd-2440-419f-9403-73f10f421359" (UID: "6e2d17bd-2440-419f-9403-73f10f421359"). InnerVolumeSpecName "kube-api-access-tnbnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:14:11.572409 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.572342 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-cluster-tls-config" pod "6e2d17bd-2440-419f-9403-73f10f421359" (UID: "6e2d17bd-2440-419f-9403-73f10f421359"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:11.578564 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.578540 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-web-config" pod "6e2d17bd-2440-419f-9403-73f10f421359" (UID: "6e2d17bd-2440-419f-9403-73f10f421359"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:11.666445 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.666351 6507 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-main-db\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:11.666445 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.666388 6507 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:11.666445 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.666400 6507 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-tls-assets\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:11.666445 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.666409 6507 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-main-tls\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:11.666445 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.666418 6507 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:11.666445 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.666427 6507 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:11.666445 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.666436 6507 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-cluster-tls-config\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:11.666445 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.666444 6507 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e2d17bd-2440-419f-9403-73f10f421359-config-out\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:11.666826 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.666452 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnbnn\" (UniqueName: \"kubernetes.io/projected/6e2d17bd-2440-419f-9403-73f10f421359-kube-api-access-tnbnn\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:11.666826 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.666481 6507 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-web-config\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:11.666826 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.666489 6507 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e2d17bd-2440-419f-9403-73f10f421359-metrics-client-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:11.666826 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.666498 6507 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:11.666826 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.666506 6507 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e2d17bd-2440-419f-9403-73f10f421359-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:11.888240 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.888204 6507 generic.go:356] "Generic (PLEG): container finished" podID="6e2d17bd-2440-419f-9403-73f10f421359" containerID="147c61fdc70a55c617acd2b28d32e2e3b32e694f38e2f2fd9ea28692cffce801" exitCode=0 Oct 29 23:14:11.888407 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.888251 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e2d17bd-2440-419f-9403-73f10f421359","Type":"ContainerDied","Data":"147c61fdc70a55c617acd2b28d32e2e3b32e694f38e2f2fd9ea28692cffce801"} Oct 29 23:14:11.888407 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.888312 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 29 23:14:11.888407 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.888317 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e2d17bd-2440-419f-9403-73f10f421359","Type":"ContainerDied","Data":"6f56629e6314704d696b28a21f2fbf244aebd8c6100286458fcd858c4d37d12c"} Oct 29 23:14:11.888407 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.888352 6507 scope.go:122] "RemoveContainer" containerID="1e5212ad899b6158f44ac73409016545410d98f0b5c9504631292948e52e86d9" Oct 29 23:14:11.896868 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.896849 6507 scope.go:122] "RemoveContainer" containerID="e01f48c65de1075865455a098a5d1241260e5af5c6f4df20a79ffd89e9be4b5e" Oct 29 23:14:11.903768 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.903753 6507 scope.go:122] "RemoveContainer" containerID="372657f190070a2029a6036dd7ad3e45a76fddc8ed6e9fdad04fecacd5b50cda" Oct 29 23:14:11.910310 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.910292 6507 scope.go:122] "RemoveContainer" containerID="147c61fdc70a55c617acd2b28d32e2e3b32e694f38e2f2fd9ea28692cffce801" Oct 29 23:14:11.916681 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.916663 6507 scope.go:122] "RemoveContainer" containerID="aea55638adf89557b3d2f1a784c52c2cda19c440a6f930fa5d3d7bb6d8cfca8a" Oct 29 23:14:11.923615 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.923597 6507 scope.go:122] "RemoveContainer" containerID="59250c65c1ead7102d5de10cea540bcec94919334a6eccb71632a3760316cdcf" Oct 29 23:14:11.931990 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.931965 6507 scope.go:122] "RemoveContainer" containerID="eafbb4d1a4d3ec0c0d9881397c0f6acbc4799761dde36166a54b45df8fd53bf1" Oct 29 23:14:11.938671 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.938655 6507 scope.go:122] "RemoveContainer" containerID="1e5212ad899b6158f44ac73409016545410d98f0b5c9504631292948e52e86d9" Oct 29 23:14:11.938972 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:11.938948 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5212ad899b6158f44ac73409016545410d98f0b5c9504631292948e52e86d9\": container with ID starting with 1e5212ad899b6158f44ac73409016545410d98f0b5c9504631292948e52e86d9 not found: ID does not exist" containerID="1e5212ad899b6158f44ac73409016545410d98f0b5c9504631292948e52e86d9" Oct 29 23:14:11.939056 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.938978 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5212ad899b6158f44ac73409016545410d98f0b5c9504631292948e52e86d9"} err="failed to get container status \"1e5212ad899b6158f44ac73409016545410d98f0b5c9504631292948e52e86d9\": rpc error: code = NotFound desc = could not find container \"1e5212ad899b6158f44ac73409016545410d98f0b5c9504631292948e52e86d9\": container with ID starting with 1e5212ad899b6158f44ac73409016545410d98f0b5c9504631292948e52e86d9 not found: ID does not exist" Oct 29 23:14:11.939056 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.939010 6507 scope.go:122] "RemoveContainer" containerID="e01f48c65de1075865455a098a5d1241260e5af5c6f4df20a79ffd89e9be4b5e" Oct 29 23:14:11.939245 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:11.939228 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e01f48c65de1075865455a098a5d1241260e5af5c6f4df20a79ffd89e9be4b5e\": container with ID starting with e01f48c65de1075865455a098a5d1241260e5af5c6f4df20a79ffd89e9be4b5e not found: ID does not exist" containerID="e01f48c65de1075865455a098a5d1241260e5af5c6f4df20a79ffd89e9be4b5e" Oct 29 23:14:11.939299 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.939251 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01f48c65de1075865455a098a5d1241260e5af5c6f4df20a79ffd89e9be4b5e"} err="failed to get container status \"e01f48c65de1075865455a098a5d1241260e5af5c6f4df20a79ffd89e9be4b5e\": rpc error: code = NotFound desc = could not find container \"e01f48c65de1075865455a098a5d1241260e5af5c6f4df20a79ffd89e9be4b5e\": container with ID starting with e01f48c65de1075865455a098a5d1241260e5af5c6f4df20a79ffd89e9be4b5e not found: ID does not exist" Oct 29 23:14:11.939299 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.939265 6507 scope.go:122] "RemoveContainer" containerID="372657f190070a2029a6036dd7ad3e45a76fddc8ed6e9fdad04fecacd5b50cda" Oct 29 23:14:11.939491 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:11.939478 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372657f190070a2029a6036dd7ad3e45a76fddc8ed6e9fdad04fecacd5b50cda\": container with ID starting with 372657f190070a2029a6036dd7ad3e45a76fddc8ed6e9fdad04fecacd5b50cda not found: ID does not exist" containerID="372657f190070a2029a6036dd7ad3e45a76fddc8ed6e9fdad04fecacd5b50cda" Oct 29 23:14:11.939537 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.939498 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372657f190070a2029a6036dd7ad3e45a76fddc8ed6e9fdad04fecacd5b50cda"} err="failed to get container status \"372657f190070a2029a6036dd7ad3e45a76fddc8ed6e9fdad04fecacd5b50cda\": rpc error: code = NotFound desc = could not find container \"372657f190070a2029a6036dd7ad3e45a76fddc8ed6e9fdad04fecacd5b50cda\": container with ID starting with 372657f190070a2029a6036dd7ad3e45a76fddc8ed6e9fdad04fecacd5b50cda not found: ID does not exist" Oct 29 23:14:11.939537 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.939511 6507 scope.go:122] "RemoveContainer" containerID="147c61fdc70a55c617acd2b28d32e2e3b32e694f38e2f2fd9ea28692cffce801" Oct 29 23:14:11.939719 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:11.939705 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147c61fdc70a55c617acd2b28d32e2e3b32e694f38e2f2fd9ea28692cffce801\": container with ID starting with 147c61fdc70a55c617acd2b28d32e2e3b32e694f38e2f2fd9ea28692cffce801 not found: ID does not exist" containerID="147c61fdc70a55c617acd2b28d32e2e3b32e694f38e2f2fd9ea28692cffce801" Oct 29 23:14:11.939761 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.939722 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147c61fdc70a55c617acd2b28d32e2e3b32e694f38e2f2fd9ea28692cffce801"} err="failed to get container status \"147c61fdc70a55c617acd2b28d32e2e3b32e694f38e2f2fd9ea28692cffce801\": rpc error: code = NotFound desc = could not find container \"147c61fdc70a55c617acd2b28d32e2e3b32e694f38e2f2fd9ea28692cffce801\": container with ID starting with 147c61fdc70a55c617acd2b28d32e2e3b32e694f38e2f2fd9ea28692cffce801 not found: ID does not exist" Oct 29 23:14:11.939761 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.939733 6507 scope.go:122] "RemoveContainer" containerID="aea55638adf89557b3d2f1a784c52c2cda19c440a6f930fa5d3d7bb6d8cfca8a" Oct 29 23:14:11.940006 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:11.939977 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea55638adf89557b3d2f1a784c52c2cda19c440a6f930fa5d3d7bb6d8cfca8a\": container with ID starting with aea55638adf89557b3d2f1a784c52c2cda19c440a6f930fa5d3d7bb6d8cfca8a not found: ID does not exist" containerID="aea55638adf89557b3d2f1a784c52c2cda19c440a6f930fa5d3d7bb6d8cfca8a" Oct 29 23:14:11.940072 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.940020 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea55638adf89557b3d2f1a784c52c2cda19c440a6f930fa5d3d7bb6d8cfca8a"} err="failed to get container status \"aea55638adf89557b3d2f1a784c52c2cda19c440a6f930fa5d3d7bb6d8cfca8a\": rpc error: code = NotFound desc = could not find container \"aea55638adf89557b3d2f1a784c52c2cda19c440a6f930fa5d3d7bb6d8cfca8a\": container with ID starting with aea55638adf89557b3d2f1a784c52c2cda19c440a6f930fa5d3d7bb6d8cfca8a not found: ID does not exist" Oct 29 23:14:11.940072 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.940043 6507 scope.go:122] "RemoveContainer" containerID="59250c65c1ead7102d5de10cea540bcec94919334a6eccb71632a3760316cdcf" Oct 29 23:14:11.940374 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:11.940355 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59250c65c1ead7102d5de10cea540bcec94919334a6eccb71632a3760316cdcf\": container with ID starting with 59250c65c1ead7102d5de10cea540bcec94919334a6eccb71632a3760316cdcf not found: ID does not exist" containerID="59250c65c1ead7102d5de10cea540bcec94919334a6eccb71632a3760316cdcf" Oct 29 23:14:11.940479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.940376 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59250c65c1ead7102d5de10cea540bcec94919334a6eccb71632a3760316cdcf"} err="failed to get container status \"59250c65c1ead7102d5de10cea540bcec94919334a6eccb71632a3760316cdcf\": rpc error: code = NotFound desc = could not find container \"59250c65c1ead7102d5de10cea540bcec94919334a6eccb71632a3760316cdcf\": container with ID starting with 59250c65c1ead7102d5de10cea540bcec94919334a6eccb71632a3760316cdcf not found: ID does not exist" Oct 29 23:14:11.940479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.940403 6507 scope.go:122] "RemoveContainer" containerID="eafbb4d1a4d3ec0c0d9881397c0f6acbc4799761dde36166a54b45df8fd53bf1" Oct 29 23:14:11.940738 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:11.940718 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eafbb4d1a4d3ec0c0d9881397c0f6acbc4799761dde36166a54b45df8fd53bf1\": container with ID starting with eafbb4d1a4d3ec0c0d9881397c0f6acbc4799761dde36166a54b45df8fd53bf1 not found: ID does not exist" containerID="eafbb4d1a4d3ec0c0d9881397c0f6acbc4799761dde36166a54b45df8fd53bf1" Oct 29 23:14:11.940804 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.940747 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eafbb4d1a4d3ec0c0d9881397c0f6acbc4799761dde36166a54b45df8fd53bf1"} err="failed to get container status \"eafbb4d1a4d3ec0c0d9881397c0f6acbc4799761dde36166a54b45df8fd53bf1\": rpc error: code = NotFound desc = could not find container \"eafbb4d1a4d3ec0c0d9881397c0f6acbc4799761dde36166a54b45df8fd53bf1\": container with ID starting with eafbb4d1a4d3ec0c0d9881397c0f6acbc4799761dde36166a54b45df8fd53bf1 not found: ID does not exist" Oct 29 23:14:11.941355 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.941339 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 23:14:11.952592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:11.952573 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 29 23:14:12.749619 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:12.749586 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e2d17bd-2440-419f-9403-73f10f421359" path="/var/lib/kubelet/pods/6e2d17bd-2440-419f-9403-73f10f421359/volumes" Oct 29 23:14:13.463284 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:13.463256 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-klctw" Oct 29 23:14:14.862359 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:14.862329 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 23:14:14.862802 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:14.862622 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="prometheus" containerID="cri-o://7feb7ca52668f5d3582ea64bbb6b6d20d0acd0b301050b221fbe584abee58722" gracePeriod=600 Oct 29 23:14:14.862802 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:14.862661 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="kube-rbac-proxy" containerID="cri-o://488b57da1b9c5521c3e976682973b7f53163694e80170705a6f27c5529ca3776" gracePeriod=600 Oct 29 23:14:14.862802 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:14.862672 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="kube-rbac-proxy-web" containerID="cri-o://f26ac97deff254d52616bbafe03846b0145321f1201a42d6eeb5dc12163bde80" gracePeriod=600 Oct 29 23:14:14.862802 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:14.862711 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="thanos-sidecar" containerID="cri-o://c39ce6033548ccafb4f26caaa178b83e737b80ba5968f1879906884bc93a8960" gracePeriod=600 Oct 29 23:14:14.863550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:14.862763 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e1cab712bd585284b0e4ef4654f239e190a4a7703c50d4c8d9ec9b19dd6d60cc" gracePeriod=600 Oct 29 23:14:14.863969 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:14.863770 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="config-reloader" containerID="cri-o://dc89d501d1ea72f9863d2f40ba87257fe8fdb40ce4985025e9635b09edf3e273" gracePeriod=600 Oct 29 23:14:15.222331 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:15.222295 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:14:15.904715 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:15.904678 6507 generic.go:356] "Generic (PLEG): container finished" podID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerID="e1cab712bd585284b0e4ef4654f239e190a4a7703c50d4c8d9ec9b19dd6d60cc" exitCode=0 Oct 29 23:14:15.904715 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:15.904702 6507 generic.go:356] "Generic (PLEG): container finished" podID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerID="488b57da1b9c5521c3e976682973b7f53163694e80170705a6f27c5529ca3776" exitCode=0 Oct 29 23:14:15.904715 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:15.904710 6507 generic.go:356] "Generic (PLEG): container finished" podID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerID="c39ce6033548ccafb4f26caaa178b83e737b80ba5968f1879906884bc93a8960" exitCode=0 Oct 29 23:14:15.904715 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:15.904716 6507 generic.go:356] "Generic (PLEG): container finished" podID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerID="dc89d501d1ea72f9863d2f40ba87257fe8fdb40ce4985025e9635b09edf3e273" exitCode=0 Oct 29 23:14:15.904715 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:15.904722 6507 generic.go:356] "Generic (PLEG): container finished" podID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerID="7feb7ca52668f5d3582ea64bbb6b6d20d0acd0b301050b221fbe584abee58722" exitCode=0 Oct 29 23:14:15.905212 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:15.904746 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc","Type":"ContainerDied","Data":"e1cab712bd585284b0e4ef4654f239e190a4a7703c50d4c8d9ec9b19dd6d60cc"} Oct 29 23:14:15.905212 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:15.904835 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc","Type":"ContainerDied","Data":"488b57da1b9c5521c3e976682973b7f53163694e80170705a6f27c5529ca3776"} Oct 29 23:14:15.905212 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:15.904864 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc","Type":"ContainerDied","Data":"c39ce6033548ccafb4f26caaa178b83e737b80ba5968f1879906884bc93a8960"} Oct 29 23:14:15.905212 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:15.904887 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc","Type":"ContainerDied","Data":"dc89d501d1ea72f9863d2f40ba87257fe8fdb40ce4985025e9635b09edf3e273"} Oct 29 23:14:15.905212 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:15.904909 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc","Type":"ContainerDied","Data":"7feb7ca52668f5d3582ea64bbb6b6d20d0acd0b301050b221fbe584abee58722"} Oct 29 23:14:16.141038 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.141006 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:16.210137 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210111 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210145 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-tls-assets\" (UniqueName: \"kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-tls-assets\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210177 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-1\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210206 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-metrics-client-ca\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210228 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210249 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-2\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210275 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-metrics-client-certs\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210314 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-trusted-ca-bundle\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210355 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-tls\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210389 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-grpc-tls\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210420 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-web-config\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-web-config\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210451 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config-out\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210513 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-kube-rbac-proxy\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210551 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-thanos-prometheus-http-client-file\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210564 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-1" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:16.210853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210579 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-serving-certs-ca-bundle\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210650 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-0\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210691 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210721 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-kube-api-access-6cxhf\" (UniqueName: \"kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-kube-api-access-6cxhf\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210727 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-metrics-client-ca" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:16.210853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210750 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-db\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.210853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210794 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-kubelet-serving-ca-bundle\") pod \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\" (UID: \"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc\") " Oct 29 23:14:16.211308 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.210863 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-2" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:16.211308 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.211145 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-trusted-ca-bundle" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:16.211308 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.211218 6507 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.211308 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.211243 6507 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-metrics-client-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.211308 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.211259 6507 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.211573 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.211208 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-kubelet-serving-ca-bundle" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:16.212758 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.212398 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-0" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:16.212975 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.212950 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-serving-certs-ca-bundle" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:16.213491 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.213437 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-db" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:16.213890 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.213852 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-kube-rbac-proxy-web" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:16.214269 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.214194 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-tls-assets" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:14:16.214269 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.214331 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config-out" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:16.214558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.214511 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:16.214710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.214690 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-thanos-sidecar-tls" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:16.215238 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.215214 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-thanos-prometheus-http-client-file" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:16.215608 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.215583 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-kube-api-access-6cxhf" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "kube-api-access-6cxhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:14:16.215857 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.215831 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-kube-rbac-proxy" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:16.216368 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.216343 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-metrics-client-certs" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:16.216625 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.216606 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-tls" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:16.216755 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.216739 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-grpc-tls" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:16.225382 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.225235 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-web-config" pod "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" (UID: "3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:16.311854 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311816 6507 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.311854 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311846 6507 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-tls-assets\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.311854 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311858 6507 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.312084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311867 6507 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-metrics-client-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.312084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311877 6507 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.312084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311887 6507 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.312084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311895 6507 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-grpc-tls\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.312084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311904 6507 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-web-config\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.312084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311912 6507 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-config-out\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.312084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311920 6507 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-kube-rbac-proxy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.312084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311928 6507 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.312084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311938 6507 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.312084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311947 6507 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.312084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311956 6507 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.312084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311965 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cxhf\" (UniqueName: \"kubernetes.io/projected/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-kube-api-access-6cxhf\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.312084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311974 6507 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-prometheus-k8s-db\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.312084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.311982 6507 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:16.810788 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.810738 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5cc6cb8466-tphkb" podUID="defafe8e-3565-464e-b2be-522d526248a4" containerName="console" containerID="cri-o://be2eaccc4c8d7c97006308ffd516c6049d44fdde0f5d35894ba6eb42dec87ee8" gracePeriod=15 Oct 29 23:14:16.911568 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.911531 6507 generic.go:356] "Generic (PLEG): container finished" podID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerID="f26ac97deff254d52616bbafe03846b0145321f1201a42d6eeb5dc12163bde80" exitCode=0 Oct 29 23:14:16.911948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.911595 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc","Type":"ContainerDied","Data":"f26ac97deff254d52616bbafe03846b0145321f1201a42d6eeb5dc12163bde80"} Oct 29 23:14:16.911948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.911624 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:16.911948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.911674 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc","Type":"ContainerDied","Data":"f47734bfb9a7143130d8cb58a4a3e3999c5b8a5ed3b82f5ae191b1c8babc1178"} Oct 29 23:14:16.911948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.911721 6507 scope.go:122] "RemoveContainer" containerID="e1cab712bd585284b0e4ef4654f239e190a4a7703c50d4c8d9ec9b19dd6d60cc" Oct 29 23:14:16.920261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.920239 6507 scope.go:122] "RemoveContainer" containerID="488b57da1b9c5521c3e976682973b7f53163694e80170705a6f27c5529ca3776" Oct 29 23:14:16.927261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.927239 6507 scope.go:122] "RemoveContainer" containerID="f26ac97deff254d52616bbafe03846b0145321f1201a42d6eeb5dc12163bde80" Oct 29 23:14:16.933314 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.933295 6507 scope.go:122] "RemoveContainer" containerID="c39ce6033548ccafb4f26caaa178b83e737b80ba5968f1879906884bc93a8960" Oct 29 23:14:16.939447 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.939429 6507 scope.go:122] "RemoveContainer" containerID="dc89d501d1ea72f9863d2f40ba87257fe8fdb40ce4985025e9635b09edf3e273" Oct 29 23:14:16.945682 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.945666 6507 scope.go:122] "RemoveContainer" containerID="7feb7ca52668f5d3582ea64bbb6b6d20d0acd0b301050b221fbe584abee58722" Oct 29 23:14:16.953591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.953571 6507 scope.go:122] "RemoveContainer" containerID="58271d5668a0a40f2a7a29fde78624521d20d0b178611b7075a8e004751862dc" Oct 29 23:14:16.959827 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.959811 6507 scope.go:122] "RemoveContainer" containerID="e1cab712bd585284b0e4ef4654f239e190a4a7703c50d4c8d9ec9b19dd6d60cc" Oct 29 23:14:16.960111 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:16.960091 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1cab712bd585284b0e4ef4654f239e190a4a7703c50d4c8d9ec9b19dd6d60cc\": container with ID starting with e1cab712bd585284b0e4ef4654f239e190a4a7703c50d4c8d9ec9b19dd6d60cc not found: ID does not exist" containerID="e1cab712bd585284b0e4ef4654f239e190a4a7703c50d4c8d9ec9b19dd6d60cc" Oct 29 23:14:16.960197 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.960120 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1cab712bd585284b0e4ef4654f239e190a4a7703c50d4c8d9ec9b19dd6d60cc"} err="failed to get container status \"e1cab712bd585284b0e4ef4654f239e190a4a7703c50d4c8d9ec9b19dd6d60cc\": rpc error: code = NotFound desc = could not find container \"e1cab712bd585284b0e4ef4654f239e190a4a7703c50d4c8d9ec9b19dd6d60cc\": container with ID starting with e1cab712bd585284b0e4ef4654f239e190a4a7703c50d4c8d9ec9b19dd6d60cc not found: ID does not exist" Oct 29 23:14:16.960197 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.960140 6507 scope.go:122] "RemoveContainer" containerID="488b57da1b9c5521c3e976682973b7f53163694e80170705a6f27c5529ca3776" Oct 29 23:14:16.960375 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:16.960357 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"488b57da1b9c5521c3e976682973b7f53163694e80170705a6f27c5529ca3776\": container with ID starting with 488b57da1b9c5521c3e976682973b7f53163694e80170705a6f27c5529ca3776 not found: ID does not exist" containerID="488b57da1b9c5521c3e976682973b7f53163694e80170705a6f27c5529ca3776" Oct 29 23:14:16.960418 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.960380 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"488b57da1b9c5521c3e976682973b7f53163694e80170705a6f27c5529ca3776"} err="failed to get container status \"488b57da1b9c5521c3e976682973b7f53163694e80170705a6f27c5529ca3776\": rpc error: code = NotFound desc = could not find container \"488b57da1b9c5521c3e976682973b7f53163694e80170705a6f27c5529ca3776\": container with ID starting with 488b57da1b9c5521c3e976682973b7f53163694e80170705a6f27c5529ca3776 not found: ID does not exist" Oct 29 23:14:16.960418 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.960396 6507 scope.go:122] "RemoveContainer" containerID="f26ac97deff254d52616bbafe03846b0145321f1201a42d6eeb5dc12163bde80" Oct 29 23:14:16.960630 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:16.960613 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26ac97deff254d52616bbafe03846b0145321f1201a42d6eeb5dc12163bde80\": container with ID starting with f26ac97deff254d52616bbafe03846b0145321f1201a42d6eeb5dc12163bde80 not found: ID does not exist" containerID="f26ac97deff254d52616bbafe03846b0145321f1201a42d6eeb5dc12163bde80" Oct 29 23:14:16.960680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.960633 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26ac97deff254d52616bbafe03846b0145321f1201a42d6eeb5dc12163bde80"} err="failed to get container status \"f26ac97deff254d52616bbafe03846b0145321f1201a42d6eeb5dc12163bde80\": rpc error: code = NotFound desc = could not find container \"f26ac97deff254d52616bbafe03846b0145321f1201a42d6eeb5dc12163bde80\": container with ID starting with f26ac97deff254d52616bbafe03846b0145321f1201a42d6eeb5dc12163bde80 not found: ID does not exist" Oct 29 23:14:16.960680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.960645 6507 scope.go:122] "RemoveContainer" containerID="c39ce6033548ccafb4f26caaa178b83e737b80ba5968f1879906884bc93a8960" Oct 29 23:14:16.960904 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:16.960881 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39ce6033548ccafb4f26caaa178b83e737b80ba5968f1879906884bc93a8960\": container with ID starting with c39ce6033548ccafb4f26caaa178b83e737b80ba5968f1879906884bc93a8960 not found: ID does not exist" containerID="c39ce6033548ccafb4f26caaa178b83e737b80ba5968f1879906884bc93a8960" Oct 29 23:14:16.960971 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.960914 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39ce6033548ccafb4f26caaa178b83e737b80ba5968f1879906884bc93a8960"} err="failed to get container status \"c39ce6033548ccafb4f26caaa178b83e737b80ba5968f1879906884bc93a8960\": rpc error: code = NotFound desc = could not find container \"c39ce6033548ccafb4f26caaa178b83e737b80ba5968f1879906884bc93a8960\": container with ID starting with c39ce6033548ccafb4f26caaa178b83e737b80ba5968f1879906884bc93a8960 not found: ID does not exist" Oct 29 23:14:16.960971 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.960939 6507 scope.go:122] "RemoveContainer" containerID="dc89d501d1ea72f9863d2f40ba87257fe8fdb40ce4985025e9635b09edf3e273" Oct 29 23:14:16.961333 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:16.961315 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc89d501d1ea72f9863d2f40ba87257fe8fdb40ce4985025e9635b09edf3e273\": container with ID starting with dc89d501d1ea72f9863d2f40ba87257fe8fdb40ce4985025e9635b09edf3e273 not found: ID does not exist" containerID="dc89d501d1ea72f9863d2f40ba87257fe8fdb40ce4985025e9635b09edf3e273" Oct 29 23:14:16.961404 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.961340 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc89d501d1ea72f9863d2f40ba87257fe8fdb40ce4985025e9635b09edf3e273"} err="failed to get container status \"dc89d501d1ea72f9863d2f40ba87257fe8fdb40ce4985025e9635b09edf3e273\": rpc error: code = NotFound desc = could not find container \"dc89d501d1ea72f9863d2f40ba87257fe8fdb40ce4985025e9635b09edf3e273\": container with ID starting with dc89d501d1ea72f9863d2f40ba87257fe8fdb40ce4985025e9635b09edf3e273 not found: ID does not exist" Oct 29 23:14:16.961404 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.961358 6507 scope.go:122] "RemoveContainer" containerID="7feb7ca52668f5d3582ea64bbb6b6d20d0acd0b301050b221fbe584abee58722" Oct 29 23:14:16.961620 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:16.961598 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7feb7ca52668f5d3582ea64bbb6b6d20d0acd0b301050b221fbe584abee58722\": container with ID starting with 7feb7ca52668f5d3582ea64bbb6b6d20d0acd0b301050b221fbe584abee58722 not found: ID does not exist" containerID="7feb7ca52668f5d3582ea64bbb6b6d20d0acd0b301050b221fbe584abee58722" Oct 29 23:14:16.961662 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.961626 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7feb7ca52668f5d3582ea64bbb6b6d20d0acd0b301050b221fbe584abee58722"} err="failed to get container status \"7feb7ca52668f5d3582ea64bbb6b6d20d0acd0b301050b221fbe584abee58722\": rpc error: code = NotFound desc = could not find container \"7feb7ca52668f5d3582ea64bbb6b6d20d0acd0b301050b221fbe584abee58722\": container with ID starting with 7feb7ca52668f5d3582ea64bbb6b6d20d0acd0b301050b221fbe584abee58722 not found: ID does not exist" Oct 29 23:14:16.961662 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.961648 6507 scope.go:122] "RemoveContainer" containerID="58271d5668a0a40f2a7a29fde78624521d20d0b178611b7075a8e004751862dc" Oct 29 23:14:16.962035 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:16.961981 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58271d5668a0a40f2a7a29fde78624521d20d0b178611b7075a8e004751862dc\": container with ID starting with 58271d5668a0a40f2a7a29fde78624521d20d0b178611b7075a8e004751862dc not found: ID does not exist" containerID="58271d5668a0a40f2a7a29fde78624521d20d0b178611b7075a8e004751862dc" Oct 29 23:14:16.962035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.962006 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58271d5668a0a40f2a7a29fde78624521d20d0b178611b7075a8e004751862dc"} err="failed to get container status \"58271d5668a0a40f2a7a29fde78624521d20d0b178611b7075a8e004751862dc\": rpc error: code = NotFound desc = could not find container \"58271d5668a0a40f2a7a29fde78624521d20d0b178611b7075a8e004751862dc\": container with ID starting with 58271d5668a0a40f2a7a29fde78624521d20d0b178611b7075a8e004751862dc not found: ID does not exist" Oct 29 23:14:16.962208 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.962113 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 23:14:16.966402 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.966378 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 23:14:16.993392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993361 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 23:14:16.993763 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993744 6507 cpu_manager.go:414] "containerMap: removing container" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="prom-label-proxy" Oct 29 23:14:16.993763 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993764 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="prom-label-proxy" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993774 6507 cpu_manager.go:414] "containerMap: removing container" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="thanos-sidecar" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993780 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="thanos-sidecar" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993787 6507 cpu_manager.go:414] "containerMap: removing container" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="kube-rbac-proxy" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993793 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="kube-rbac-proxy" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993803 6507 cpu_manager.go:414] "containerMap: removing container" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="kube-rbac-proxy-thanos" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993807 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="kube-rbac-proxy-thanos" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993814 6507 cpu_manager.go:414] "containerMap: removing container" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="config-reloader" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993819 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="config-reloader" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993829 6507 cpu_manager.go:414] "containerMap: removing container" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="init-config-reloader" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993834 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="init-config-reloader" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993842 6507 cpu_manager.go:414] "containerMap: removing container" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="kube-rbac-proxy-web" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993846 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="kube-rbac-proxy-web" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993852 6507 cpu_manager.go:414] "containerMap: removing container" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="prometheus" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993857 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="prometheus" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993866 6507 cpu_manager.go:414] "containerMap: removing container" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="kube-rbac-proxy" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993870 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="kube-rbac-proxy" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993876 6507 cpu_manager.go:414] "containerMap: removing container" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="init-config-reloader" Oct 29 23:14:16.993877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993880 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="init-config-reloader" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993887 6507 cpu_manager.go:414] "containerMap: removing container" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="kube-rbac-proxy-web" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993891 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="kube-rbac-proxy-web" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993896 6507 cpu_manager.go:414] "containerMap: removing container" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="alertmanager" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993901 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="alertmanager" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993908 6507 cpu_manager.go:414] "containerMap: removing container" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="config-reloader" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993911 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="config-reloader" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993918 6507 cpu_manager.go:414] "containerMap: removing container" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="kube-rbac-proxy-metric" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993922 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="kube-rbac-proxy-metric" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993960 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="kube-rbac-proxy-metric" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993968 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="prom-label-proxy" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993974 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="thanos-sidecar" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993979 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="kube-rbac-proxy" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993985 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="kube-rbac-proxy-thanos" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993991 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="config-reloader" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.993996 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="init-config-reloader" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.994002 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="kube-rbac-proxy-web" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.994006 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="prometheus" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.994011 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="kube-rbac-proxy" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.994016 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="init-config-reloader" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.994021 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="kube-rbac-proxy-web" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.994026 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2d17bd-2440-419f-9403-73f10f421359" containerName="alertmanager" Oct 29 23:14:16.994548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:16.994031 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" containerName="config-reloader" Oct 29 23:14:17.027593 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.027513 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.037394 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.037359 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 23:14:17.068190 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.067206 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 29 23:14:17.068190 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.067241 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-59hcs\"" Oct 29 23:14:17.068190 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.067302 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ejptm05c9424c\"" Oct 29 23:14:17.068190 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.067353 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 29 23:14:17.068190 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.067492 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 29 23:14:17.068190 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.067569 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 29 23:14:17.068190 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.067622 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 29 23:14:17.068190 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.067632 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 29 23:14:17.068190 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.067707 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 29 23:14:17.068190 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.067786 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 29 23:14:17.068190 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.068195 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 29 23:14:17.113641 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.113607 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:14:17.119495 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.119466 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 29 23:14:17.119495 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.119474 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 29 23:14:17.119671 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.119570 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 29 23:14:17.119671 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.119614 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 29 23:14:17.119671 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.119646 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.119801 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.119676 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.119801 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.119783 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.119868 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.119818 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.119868 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.119840 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.119949 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.119914 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.119949 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.119937 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.120035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.119966 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-config\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.120035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.119985 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-web-config\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.120035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.120017 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.120260 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.120178 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.120420 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.120403 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7rdm\" (UniqueName: \"kubernetes.io/projected/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-kube-api-access-w7rdm\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.120482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.120448 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.120530 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.120495 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-config-out\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.120530 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.120519 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.120636 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.120535 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.120678 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.120629 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.120678 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.120672 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.120740 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.120719 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.120780 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.120736 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.131803 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.131779 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 29 23:14:17.131905 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.131865 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 29 23:14:17.221368 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221327 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca\") pod \"defafe8e-3565-464e-b2be-522d526248a4\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " Oct 29 23:14:17.221547 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221410 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert\") pod \"defafe8e-3565-464e-b2be-522d526248a4\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " Oct 29 23:14:17.221547 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221451 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config\") pod \"defafe8e-3565-464e-b2be-522d526248a4\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " Oct 29 23:14:17.221547 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221515 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defafe8e-3565-464e-b2be-522d526248a4-tmp\" (UniqueName: \"kubernetes.io/empty-dir/defafe8e-3565-464e-b2be-522d526248a4-tmp\") pod \"defafe8e-3565-464e-b2be-522d526248a4\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " Oct 29 23:14:17.221547 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221540 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert\") pod \"defafe8e-3565-464e-b2be-522d526248a4\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " Oct 29 23:14:17.221764 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221560 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle\") pod \"defafe8e-3565-464e-b2be-522d526248a4\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " Oct 29 23:14:17.221764 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221588 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk\" (UniqueName: \"kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk\") pod \"defafe8e-3565-464e-b2be-522d526248a4\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " Oct 29 23:14:17.221764 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221627 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config\") pod \"defafe8e-3565-464e-b2be-522d526248a4\" (UID: \"defafe8e-3565-464e-b2be-522d526248a4\") " Oct 29 23:14:17.221764 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221703 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.221764 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221732 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.221764 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221758 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221782 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221812 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221844 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221868 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221911 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221910 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca" pod "defafe8e-3565-464e-b2be-522d526248a4" (UID: "defafe8e-3565-464e-b2be-522d526248a4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:17.222045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221935 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221914 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert" pod "defafe8e-3565-464e-b2be-522d526248a4" (UID: "defafe8e-3565-464e-b2be-522d526248a4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:17.222045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221987 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-config\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.222017 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-web-config\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.222046 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222594 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.222068 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222594 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.222096 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7rdm\" (UniqueName: \"kubernetes.io/projected/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-kube-api-access-w7rdm\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222594 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.222128 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222594 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.222161 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-config-out\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222594 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.222185 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222594 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.222210 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222594 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.222242 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222594 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.222275 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.222594 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.222293 6507 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 23:14:17.222594 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.222346 6507 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-service-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:17.222594 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.222366 6507 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-oauth-serving-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:17.223138 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.222883 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.223138 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.223032 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.224073 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.223976 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.225275 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.225249 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.225372 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.221987 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle" pod "defafe8e-3565-464e-b2be-522d526248a4" (UID: "defafe8e-3565-464e-b2be-522d526248a4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:17.225372 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.222172 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defafe8e-3565-464e-b2be-522d526248a4-tmp" pod "defafe8e-3565-464e-b2be-522d526248a4" (UID: "defafe8e-3565-464e-b2be-522d526248a4"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:17.225372 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.224609 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config" pod "defafe8e-3565-464e-b2be-522d526248a4" (UID: "defafe8e-3565-464e-b2be-522d526248a4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:17.225372 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.224682 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk" pod "defafe8e-3565-464e-b2be-522d526248a4" (UID: "defafe8e-3565-464e-b2be-522d526248a4"). InnerVolumeSpecName "kube-api-access-jnbnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:14:17.225372 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.224979 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config" pod "defafe8e-3565-464e-b2be-522d526248a4" (UID: "defafe8e-3565-464e-b2be-522d526248a4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:17.225653 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.225625 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.225707 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.225665 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.226185 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.226161 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.226357 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.226328 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.226943 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.226917 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.228018 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.227277 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.228018 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.227635 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.228018 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.227805 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.228018 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.227855 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.228018 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.227963 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.228304 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.228143 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-config\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.228304 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.228143 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-web-config\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.228585 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.228558 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert" pod "defafe8e-3565-464e-b2be-522d526248a4" (UID: "defafe8e-3565-464e-b2be-522d526248a4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:17.228585 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.228577 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-config-out\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.229265 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.229247 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.229797 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.229776 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.246395 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.246375 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7rdm\" (UniqueName: \"kubernetes.io/projected/2c95f4b0-5da7-4d43-ab9e-c55da3688e89-kube-api-access-w7rdm\") pod \"prometheus-k8s-0\" (UID: \"2c95f4b0-5da7-4d43-ab9e-c55da3688e89\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.323598 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.323525 6507 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-serving-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:17.323598 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.323555 6507 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-trusted-ca-bundle\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:17.323598 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.323566 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnbnk\" (UniqueName: \"kubernetes.io/projected/defafe8e-3565-464e-b2be-522d526248a4-kube-api-access-jnbnk\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:17.323598 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.323577 6507 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/defafe8e-3565-464e-b2be-522d526248a4-console-config\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:17.323598 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.323585 6507 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/defafe8e-3565-464e-b2be-522d526248a4-console-oauth-config\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:17.323598 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.323594 6507 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/defafe8e-3565-464e-b2be-522d526248a4-tmp\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:14:17.464382 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.464336 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:17.796387 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.796349 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 23:14:17.800757 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:14:17.800731 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c95f4b0_5da7_4d43_ab9e_c55da3688e89.slice/crio-282262dcb7bad49bdc6b9687153be736560f617c15d79f1bf8048374857f48b2 WatchSource:0}: Error finding container 282262dcb7bad49bdc6b9687153be736560f617c15d79f1bf8048374857f48b2: Status 404 returned error can't find the container with id 282262dcb7bad49bdc6b9687153be736560f617c15d79f1bf8048374857f48b2 Oct 29 23:14:17.917681 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.917644 6507 generic.go:356] "Generic (PLEG): container finished" podID="2c95f4b0-5da7-4d43-ab9e-c55da3688e89" containerID="13d6dd2df9cf16a6d5830bf748e4068b58cea16781d8059b97f709925b2d3526" exitCode=0 Oct 29 23:14:17.918181 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.917729 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2c95f4b0-5da7-4d43-ab9e-c55da3688e89","Type":"ContainerDied","Data":"13d6dd2df9cf16a6d5830bf748e4068b58cea16781d8059b97f709925b2d3526"} Oct 29 23:14:17.918181 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.917803 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2c95f4b0-5da7-4d43-ab9e-c55da3688e89","Type":"ContainerStarted","Data":"282262dcb7bad49bdc6b9687153be736560f617c15d79f1bf8048374857f48b2"} Oct 29 23:14:17.919072 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.919047 6507 generic.go:356] "Generic (PLEG): container finished" podID="defafe8e-3565-464e-b2be-522d526248a4" containerID="be2eaccc4c8d7c97006308ffd516c6049d44fdde0f5d35894ba6eb42dec87ee8" exitCode=0 Oct 29 23:14:17.919170 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.919088 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc6cb8466-tphkb" Oct 29 23:14:17.919170 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.919095 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc6cb8466-tphkb" event={"ID":"defafe8e-3565-464e-b2be-522d526248a4","Type":"ContainerDied","Data":"be2eaccc4c8d7c97006308ffd516c6049d44fdde0f5d35894ba6eb42dec87ee8"} Oct 29 23:14:17.919170 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.919136 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc6cb8466-tphkb" event={"ID":"defafe8e-3565-464e-b2be-522d526248a4","Type":"ContainerDied","Data":"66c61446fb25e53f3d0ae67b70a4dbd35e26e55bdc8fba9858feb4cd843c4d7b"} Oct 29 23:14:17.919170 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.919161 6507 scope.go:122] "RemoveContainer" containerID="be2eaccc4c8d7c97006308ffd516c6049d44fdde0f5d35894ba6eb42dec87ee8" Oct 29 23:14:17.927358 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.927341 6507 scope.go:122] "RemoveContainer" containerID="be2eaccc4c8d7c97006308ffd516c6049d44fdde0f5d35894ba6eb42dec87ee8" Oct 29 23:14:17.927688 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:17.927667 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2eaccc4c8d7c97006308ffd516c6049d44fdde0f5d35894ba6eb42dec87ee8\": container with ID starting with be2eaccc4c8d7c97006308ffd516c6049d44fdde0f5d35894ba6eb42dec87ee8 not found: ID does not exist" containerID="be2eaccc4c8d7c97006308ffd516c6049d44fdde0f5d35894ba6eb42dec87ee8" Oct 29 23:14:17.927758 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:17.927698 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2eaccc4c8d7c97006308ffd516c6049d44fdde0f5d35894ba6eb42dec87ee8"} err="failed to get container status \"be2eaccc4c8d7c97006308ffd516c6049d44fdde0f5d35894ba6eb42dec87ee8\": rpc error: code = NotFound desc = could not find container \"be2eaccc4c8d7c97006308ffd516c6049d44fdde0f5d35894ba6eb42dec87ee8\": container with ID starting with be2eaccc4c8d7c97006308ffd516c6049d44fdde0f5d35894ba6eb42dec87ee8 not found: ID does not exist" Oct 29 23:14:18.003235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:18.003206 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cc6cb8466-tphkb"] Oct 29 23:14:18.009421 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:18.009390 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cc6cb8466-tphkb"] Oct 29 23:14:18.749611 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:18.749579 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc" path="/var/lib/kubelet/pods/3ec1c1e5-dfa8-4b8b-bb97-e9bd48427ffc/volumes" Oct 29 23:14:18.750225 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:18.750212 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="defafe8e-3565-464e-b2be-522d526248a4" path="/var/lib/kubelet/pods/defafe8e-3565-464e-b2be-522d526248a4/volumes" Oct 29 23:14:18.925790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:18.925754 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2c95f4b0-5da7-4d43-ab9e-c55da3688e89","Type":"ContainerStarted","Data":"cfc6f301dae0095e96728fed9da0ecb6468e2a1137cd90901cee5b41fe9a1d10"} Oct 29 23:14:18.926214 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:18.925830 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2c95f4b0-5da7-4d43-ab9e-c55da3688e89","Type":"ContainerStarted","Data":"4e65d2b239f9b4a88161dbf2879ba42243b51c3d23384a385c5cc431e5b2db92"} Oct 29 23:14:18.926214 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:18.925863 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2c95f4b0-5da7-4d43-ab9e-c55da3688e89","Type":"ContainerStarted","Data":"ff59cb3ba109b8ba63df7e0039318703acd6a9da4bb34a344a712745f7226a81"} Oct 29 23:14:18.926214 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:18.925892 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2c95f4b0-5da7-4d43-ab9e-c55da3688e89","Type":"ContainerStarted","Data":"318ca928623bf1754a0963ae8df985cf03342493efd095dca7658e153f356115"} Oct 29 23:14:18.926214 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:18.925921 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2c95f4b0-5da7-4d43-ab9e-c55da3688e89","Type":"ContainerStarted","Data":"8b124223c8e2320b40a32ebd60c5d1a658546dce5cce292c4e26b501db213549"} Oct 29 23:14:18.926214 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:18.925954 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2c95f4b0-5da7-4d43-ab9e-c55da3688e89","Type":"ContainerStarted","Data":"54df75e66a2066f4831b8e126798c4ea2e51824f93dea5e3ef95b146f881613d"} Oct 29 23:14:18.985930 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:18.985879 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.967261026 podStartE2EDuration="2.985866633s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.605607ms" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:18.984853775 +0000 UTC m=+28.347858499" watchObservedRunningTime="2026-10-29 23:14:18.985866633 +0000 UTC m=+28.348871350" Oct 29 23:14:19.760655 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:19.760621 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-69jbt" Oct 29 23:14:22.464960 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:22.464916 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:45.222015 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:14:45.221975 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:14:46.055258 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:46.055224 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:46.070796 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:46.070764 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:14:47.032033 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:14:47.032000 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 23:15:15.221775 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:15:15.221739 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:15:45.223902 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:15:45.223821 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:16:15.222371 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:16:15.222330 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:16:36.595121 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:36.595086 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-7smhv"] Oct 29 23:16:36.595696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:36.595361 6507 cpu_manager.go:414] "containerMap: removing container" podUID="defafe8e-3565-464e-b2be-522d526248a4" containerName="console" Oct 29 23:16:36.595696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:36.595372 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="defafe8e-3565-464e-b2be-522d526248a4" containerName="console" Oct 29 23:16:36.595696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:36.595425 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="defafe8e-3565-464e-b2be-522d526248a4" containerName="console" Oct 29 23:16:36.597045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:36.597018 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-7smhv" Oct 29 23:16:36.621797 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:36.621763 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 29 23:16:36.621945 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:36.621875 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5g4qd\"" Oct 29 23:16:36.621945 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:36.621918 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 29 23:16:36.622418 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:36.622396 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-7smhv"] Oct 29 23:16:36.664612 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:36.664579 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2shr7\" (UniqueName: \"kubernetes.io/projected/373f7105-4b1c-41e2-9a7c-f43c7e9ecea0-kube-api-access-2shr7\") pod \"cert-manager-597b95cb84-7smhv\" (UID: \"373f7105-4b1c-41e2-9a7c-f43c7e9ecea0\") " pod="cert-manager/cert-manager-597b95cb84-7smhv" Oct 29 23:16:36.664612 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:36.664615 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/373f7105-4b1c-41e2-9a7c-f43c7e9ecea0-bound-sa-token\") pod \"cert-manager-597b95cb84-7smhv\" (UID: \"373f7105-4b1c-41e2-9a7c-f43c7e9ecea0\") " pod="cert-manager/cert-manager-597b95cb84-7smhv" Oct 29 23:16:36.765231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:36.765199 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2shr7\" (UniqueName: \"kubernetes.io/projected/373f7105-4b1c-41e2-9a7c-f43c7e9ecea0-kube-api-access-2shr7\") pod \"cert-manager-597b95cb84-7smhv\" (UID: \"373f7105-4b1c-41e2-9a7c-f43c7e9ecea0\") " pod="cert-manager/cert-manager-597b95cb84-7smhv" Oct 29 23:16:36.765231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:36.765238 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/373f7105-4b1c-41e2-9a7c-f43c7e9ecea0-bound-sa-token\") pod \"cert-manager-597b95cb84-7smhv\" (UID: \"373f7105-4b1c-41e2-9a7c-f43c7e9ecea0\") " pod="cert-manager/cert-manager-597b95cb84-7smhv" Oct 29 23:16:36.787391 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:36.787355 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/373f7105-4b1c-41e2-9a7c-f43c7e9ecea0-bound-sa-token\") pod \"cert-manager-597b95cb84-7smhv\" (UID: \"373f7105-4b1c-41e2-9a7c-f43c7e9ecea0\") " pod="cert-manager/cert-manager-597b95cb84-7smhv" Oct 29 23:16:36.788874 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:36.788854 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2shr7\" (UniqueName: \"kubernetes.io/projected/373f7105-4b1c-41e2-9a7c-f43c7e9ecea0-kube-api-access-2shr7\") pod \"cert-manager-597b95cb84-7smhv\" (UID: \"373f7105-4b1c-41e2-9a7c-f43c7e9ecea0\") " pod="cert-manager/cert-manager-597b95cb84-7smhv" Oct 29 23:16:36.916092 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:36.915984 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-7smhv" Oct 29 23:16:37.208250 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:37.208207 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-7smhv"] Oct 29 23:16:37.212855 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:16:37.212815 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod373f7105_4b1c_41e2_9a7c_f43c7e9ecea0.slice/crio-1026548797b343500b6bda8d7569354aaa4482c9012f8020791b5d9d2a852d92 WatchSource:0}: Error finding container 1026548797b343500b6bda8d7569354aaa4482c9012f8020791b5d9d2a852d92: Status 404 returned error can't find the container with id 1026548797b343500b6bda8d7569354aaa4482c9012f8020791b5d9d2a852d92 Oct 29 23:16:37.215067 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:37.215046 6507 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:16:37.371526 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:37.371493 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-7smhv" event={"ID":"373f7105-4b1c-41e2-9a7c-f43c7e9ecea0","Type":"ContainerStarted","Data":"1026548797b343500b6bda8d7569354aaa4482c9012f8020791b5d9d2a852d92"} Oct 29 23:16:40.383733 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:40.383699 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-7smhv" event={"ID":"373f7105-4b1c-41e2-9a7c-f43c7e9ecea0","Type":"ContainerStarted","Data":"7348e3fb541cf5cee25ae865f2004ba6c02bb0d4d8b0293baa1c3ebb1e3f0b03"} Oct 29 23:16:40.419215 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:40.419108 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-7smhv" podStartSLOduration=1.489097985 podStartE2EDuration="4.41909579s" totalImagesPullingTime="2.929997805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:40.419063878 +0000 UTC m=+169.782068600" watchObservedRunningTime="2026-10-29 23:16:40.41909579 +0000 UTC m=+169.782100509" Oct 29 23:16:45.222312 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:16:45.222270 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:16:59.183748 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.183691 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76"] Oct 29 23:16:59.189201 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.189169 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.223847 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.223808 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aee652fc-2f03-42d6-8ad9-ef6169e20f95-tmp\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.224032 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.223918 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aee652fc-2f03-42d6-8ad9-ef6169e20f95-metrics-cert\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.224032 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.223985 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aee652fc-2f03-42d6-8ad9-ef6169e20f95-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.224032 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.224024 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aee652fc-2f03-42d6-8ad9-ef6169e20f95-webhook-cert\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.224224 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.224041 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aee652fc-2f03-42d6-8ad9-ef6169e20f95-manifests\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.224224 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.224068 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hnw\" (UniqueName: \"kubernetes.io/projected/aee652fc-2f03-42d6-8ad9-ef6169e20f95-kube-api-access-69hnw\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.225534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.225511 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76"] Oct 29 23:16:59.287256 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.287219 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 29 23:16:59.287570 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.287320 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 29 23:16:59.287570 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.287354 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n9h54\"" Oct 29 23:16:59.287570 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.287415 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 29 23:16:59.287570 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.287574 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 29 23:16:59.287769 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.287580 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 29 23:16:59.325295 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.325255 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hnw\" (UniqueName: \"kubernetes.io/projected/aee652fc-2f03-42d6-8ad9-ef6169e20f95-kube-api-access-69hnw\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.325295 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.325300 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aee652fc-2f03-42d6-8ad9-ef6169e20f95-tmp\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.325564 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.325333 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aee652fc-2f03-42d6-8ad9-ef6169e20f95-metrics-cert\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.325564 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.325364 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aee652fc-2f03-42d6-8ad9-ef6169e20f95-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.325564 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.325390 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aee652fc-2f03-42d6-8ad9-ef6169e20f95-webhook-cert\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.325564 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.325414 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aee652fc-2f03-42d6-8ad9-ef6169e20f95-manifests\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.325914 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.325887 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aee652fc-2f03-42d6-8ad9-ef6169e20f95-manifests\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.328005 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.327974 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aee652fc-2f03-42d6-8ad9-ef6169e20f95-tmp\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.328130 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.328052 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aee652fc-2f03-42d6-8ad9-ef6169e20f95-metrics-cert\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.328130 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.328056 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aee652fc-2f03-42d6-8ad9-ef6169e20f95-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.328227 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.328132 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aee652fc-2f03-42d6-8ad9-ef6169e20f95-webhook-cert\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.344073 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.344042 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hnw\" (UniqueName: \"kubernetes.io/projected/aee652fc-2f03-42d6-8ad9-ef6169e20f95-kube-api-access-69hnw\") pod \"opendatahub-operator-controller-manager-bb7fb9d97-7bs76\" (UID: \"aee652fc-2f03-42d6-8ad9-ef6169e20f95\") " pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.503908 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.503874 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:16:59.913892 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:16:59.913864 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee652fc_2f03_42d6_8ad9_ef6169e20f95.slice/crio-fd01e0d3d5a1e046348cee1c5e0d1df5545c82c0a7f2a919e077c243179224a2 WatchSource:0}: Error finding container fd01e0d3d5a1e046348cee1c5e0d1df5545c82c0a7f2a919e077c243179224a2: Status 404 returned error can't find the container with id fd01e0d3d5a1e046348cee1c5e0d1df5545c82c0a7f2a919e077c243179224a2 Oct 29 23:16:59.920324 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:16:59.920301 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76"] Oct 29 23:17:00.454539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:00.454498 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" event={"ID":"aee652fc-2f03-42d6-8ad9-ef6169e20f95","Type":"ContainerStarted","Data":"fd01e0d3d5a1e046348cee1c5e0d1df5545c82c0a7f2a919e077c243179224a2"} Oct 29 23:17:03.465917 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:03.465882 6507 generic.go:356] "Generic (PLEG): container finished" podID="aee652fc-2f03-42d6-8ad9-ef6169e20f95" containerID="8aec5d48e4b6e16e7f833b51890657021e031c4bd1b0d30f46a644218dc98a60" exitCode=0 Oct 29 23:17:03.466312 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:03.465947 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" event={"ID":"aee652fc-2f03-42d6-8ad9-ef6169e20f95","Type":"ContainerDied","Data":"8aec5d48e4b6e16e7f833b51890657021e031c4bd1b0d30f46a644218dc98a60"} Oct 29 23:17:04.474880 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:04.474846 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" event={"ID":"aee652fc-2f03-42d6-8ad9-ef6169e20f95","Type":"ContainerStarted","Data":"b549039e913d3865c62a781b7d197517251be9ef38fd23e17de0bce100efe18c"} Oct 29 23:17:04.475273 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:04.474961 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:17:04.552233 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:04.552182 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" podStartSLOduration=1.911881988 podStartE2EDuration="5.5521694s" totalImagesPullingTime="3.154856281s" totalInitContainerRuntime="485.431131ms" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:04.540756464 +0000 UTC m=+193.903761186" watchObservedRunningTime="2026-10-29 23:17:04.5521694 +0000 UTC m=+193.915174123" Oct 29 23:17:15.223049 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:17:15.223009 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:17:15.481034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:15.480963 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" Oct 29 23:17:19.176838 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.176802 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv"] Oct 29 23:17:19.183148 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.183114 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv" Oct 29 23:17:19.200614 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.200589 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv"] Oct 29 23:17:19.211527 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.211498 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-m5slx\"" Oct 29 23:17:19.211778 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.211747 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 29 23:17:19.211778 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.211774 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 29 23:17:19.211976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.211851 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 29 23:17:19.211976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.211887 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 29 23:17:19.284473 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.284437 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smmzj\" (UniqueName: \"kubernetes.io/projected/3f783198-516a-4ada-90be-3f53da4e8cb1-kube-api-access-smmzj\") pod \"kube-auth-proxy-7cbcb88b9-dbxcv\" (UID: \"3f783198-516a-4ada-90be-3f53da4e8cb1\") " pod="openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv" Oct 29 23:17:19.284630 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.284497 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f783198-516a-4ada-90be-3f53da4e8cb1-tmp\") pod \"kube-auth-proxy-7cbcb88b9-dbxcv\" (UID: \"3f783198-516a-4ada-90be-3f53da4e8cb1\") " pod="openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv" Oct 29 23:17:19.284670 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.284625 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f783198-516a-4ada-90be-3f53da4e8cb1-tls-certs\") pod \"kube-auth-proxy-7cbcb88b9-dbxcv\" (UID: \"3f783198-516a-4ada-90be-3f53da4e8cb1\") " pod="openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv" Oct 29 23:17:19.385541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.385506 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f783198-516a-4ada-90be-3f53da4e8cb1-tls-certs\") pod \"kube-auth-proxy-7cbcb88b9-dbxcv\" (UID: \"3f783198-516a-4ada-90be-3f53da4e8cb1\") " pod="openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv" Oct 29 23:17:19.385705 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.385557 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smmzj\" (UniqueName: \"kubernetes.io/projected/3f783198-516a-4ada-90be-3f53da4e8cb1-kube-api-access-smmzj\") pod \"kube-auth-proxy-7cbcb88b9-dbxcv\" (UID: \"3f783198-516a-4ada-90be-3f53da4e8cb1\") " pod="openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv" Oct 29 23:17:19.385705 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.385586 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f783198-516a-4ada-90be-3f53da4e8cb1-tmp\") pod \"kube-auth-proxy-7cbcb88b9-dbxcv\" (UID: \"3f783198-516a-4ada-90be-3f53da4e8cb1\") " pod="openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv" Oct 29 23:17:19.387930 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.387907 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f783198-516a-4ada-90be-3f53da4e8cb1-tmp\") pod \"kube-auth-proxy-7cbcb88b9-dbxcv\" (UID: \"3f783198-516a-4ada-90be-3f53da4e8cb1\") " pod="openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv" Oct 29 23:17:19.388172 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.388150 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f783198-516a-4ada-90be-3f53da4e8cb1-tls-certs\") pod \"kube-auth-proxy-7cbcb88b9-dbxcv\" (UID: \"3f783198-516a-4ada-90be-3f53da4e8cb1\") " pod="openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv" Oct 29 23:17:19.406107 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.406072 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smmzj\" (UniqueName: \"kubernetes.io/projected/3f783198-516a-4ada-90be-3f53da4e8cb1-kube-api-access-smmzj\") pod \"kube-auth-proxy-7cbcb88b9-dbxcv\" (UID: \"3f783198-516a-4ada-90be-3f53da4e8cb1\") " pod="openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv" Oct 29 23:17:19.493887 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.493834 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv" Oct 29 23:17:19.790319 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:19.790229 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv"] Oct 29 23:17:19.795140 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:17:19.795108 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f783198_516a_4ada_90be_3f53da4e8cb1.slice/crio-3dcbcacc64fbabf5739ec0074fd6d5f6009587f12aa23d9acaaf3c6d0b4b9f25 WatchSource:0}: Error finding container 3dcbcacc64fbabf5739ec0074fd6d5f6009587f12aa23d9acaaf3c6d0b4b9f25: Status 404 returned error can't find the container with id 3dcbcacc64fbabf5739ec0074fd6d5f6009587f12aa23d9acaaf3c6d0b4b9f25 Oct 29 23:17:20.529926 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:20.529877 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv" event={"ID":"3f783198-516a-4ada-90be-3f53da4e8cb1","Type":"ContainerStarted","Data":"3dcbcacc64fbabf5739ec0074fd6d5f6009587f12aa23d9acaaf3c6d0b4b9f25"} Oct 29 23:17:22.539145 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:22.539066 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv" event={"ID":"3f783198-516a-4ada-90be-3f53da4e8cb1","Type":"ContainerStarted","Data":"2b94bb1b997743dfb6b97f278a4cfb585a699fb779ba51c76f5a805d574cb628"} Oct 29 23:17:22.578004 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:22.577947 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cbcb88b9-dbxcv" podStartSLOduration=1.179921815 podStartE2EDuration="3.577931005s" totalImagesPullingTime="2.39800919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:22.577155658 +0000 UTC m=+211.940160381" watchObservedRunningTime="2026-10-29 23:17:22.577931005 +0000 UTC m=+211.940935725" Oct 29 23:17:25.910201 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:25.910170 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l"] Oct 29 23:17:25.918346 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:25.918324 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:25.935369 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:25.935350 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l"] Oct 29 23:17:25.936914 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:25.936891 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4xh7p\"" Oct 29 23:17:25.937032 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:25.936952 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 29 23:17:26.043191 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.043158 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ca8769-66ad-496c-aed6-e91aed841fa7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.043191 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.043190 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2ca8769-66ad-496c-aed6-e91aed841fa7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.043385 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.043225 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2ca8769-66ad-496c-aed6-e91aed841fa7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.043385 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.043336 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2ca8769-66ad-496c-aed6-e91aed841fa7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.043640 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.043381 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2ca8769-66ad-496c-aed6-e91aed841fa7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.043640 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.043453 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2ca8769-66ad-496c-aed6-e91aed841fa7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.043640 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.043519 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ca8769-66ad-496c-aed6-e91aed841fa7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.043640 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.043541 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2ca8769-66ad-496c-aed6-e91aed841fa7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.043640 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.043558 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8lrz\" (UniqueName: \"kubernetes.io/projected/d2ca8769-66ad-496c-aed6-e91aed841fa7-kube-api-access-n8lrz\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.144050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.144019 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2ca8769-66ad-496c-aed6-e91aed841fa7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.144050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.144060 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2ca8769-66ad-496c-aed6-e91aed841fa7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.144398 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.144100 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2ca8769-66ad-496c-aed6-e91aed841fa7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.144398 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.144129 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2ca8769-66ad-496c-aed6-e91aed841fa7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.144398 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.144163 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ca8769-66ad-496c-aed6-e91aed841fa7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.144398 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.144195 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2ca8769-66ad-496c-aed6-e91aed841fa7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.144398 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.144218 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8lrz\" (UniqueName: \"kubernetes.io/projected/d2ca8769-66ad-496c-aed6-e91aed841fa7-kube-api-access-n8lrz\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.144398 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.144262 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ca8769-66ad-496c-aed6-e91aed841fa7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.144398 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.144283 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2ca8769-66ad-496c-aed6-e91aed841fa7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.144845 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.144663 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2ca8769-66ad-496c-aed6-e91aed841fa7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.144845 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.144775 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ca8769-66ad-496c-aed6-e91aed841fa7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.144845 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.144810 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2ca8769-66ad-496c-aed6-e91aed841fa7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.144968 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.144919 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2ca8769-66ad-496c-aed6-e91aed841fa7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.145172 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.145154 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2ca8769-66ad-496c-aed6-e91aed841fa7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.146545 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.146523 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2ca8769-66ad-496c-aed6-e91aed841fa7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.146724 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.146706 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2ca8769-66ad-496c-aed6-e91aed841fa7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.163953 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.163899 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2ca8769-66ad-496c-aed6-e91aed841fa7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.164158 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.164138 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8lrz\" (UniqueName: \"kubernetes.io/projected/d2ca8769-66ad-496c-aed6-e91aed841fa7-kube-api-access-n8lrz\") pod \"data-science-gateway-data-science-gateway-class-557c4f746fxgh9l\" (UID: \"d2ca8769-66ad-496c-aed6-e91aed841fa7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.230421 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.230390 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:26.549411 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:26.549372 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l"] Oct 29 23:17:26.554552 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:17:26.554522 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ca8769_66ad_496c_aed6_e91aed841fa7.slice/crio-58f371ccd442cb2a6d31d7277f47af8e310feed5615ce3bbc2e8fc6db864841d WatchSource:0}: Error finding container 58f371ccd442cb2a6d31d7277f47af8e310feed5615ce3bbc2e8fc6db864841d: Status 404 returned error can't find the container with id 58f371ccd442cb2a6d31d7277f47af8e310feed5615ce3bbc2e8fc6db864841d Oct 29 23:17:27.559647 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:27.559601 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" event={"ID":"d2ca8769-66ad-496c-aed6-e91aed841fa7","Type":"ContainerStarted","Data":"58f371ccd442cb2a6d31d7277f47af8e310feed5615ce3bbc2e8fc6db864841d"} Oct 29 23:17:29.141715 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:29.141682 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:17:29.141982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:29.141749 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:17:29.141982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:29.141779 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:17:29.568528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:29.568488 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" event={"ID":"d2ca8769-66ad-496c-aed6-e91aed841fa7","Type":"ContainerStarted","Data":"1b37b3893bf83bae83a80060cdcfe6abad0c644a5ff343286b4d939ececc7f63"} Oct 29 23:17:29.618474 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:29.618421 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" podStartSLOduration=2.033866736 podStartE2EDuration="4.618408219s" totalImagesPullingTime="2.584541483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:29.61777382 +0000 UTC m=+218.980778542" watchObservedRunningTime="2026-10-29 23:17:29.618408219 +0000 UTC m=+218.981412942" Oct 29 23:17:30.231473 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:30.231430 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:30.236565 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:30.236528 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:30.572484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:30.572379 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:30.573365 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:17:30.573341 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-557c4f746fxgh9l" Oct 29 23:17:45.222332 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:17:45.222292 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:18:15.223018 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:18:15.222979 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:18:21.329982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.329938 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v"] Oct 29 23:18:21.339782 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.339750 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.365356 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.365317 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 29 23:18:21.365356 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.365317 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 29 23:18:21.365356 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.365317 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 29 23:18:21.365632 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.365401 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fms52\"" Oct 29 23:18:21.366519 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.366495 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v"] Oct 29 23:18:21.382232 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.382205 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.382343 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.382239 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.382382 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.382331 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.382434 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.382416 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.382490 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.382444 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxwqn\" (UniqueName: \"kubernetes.io/projected/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-kube-api-access-mxwqn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.382524 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.382487 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.382524 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.382505 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.483260 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.483211 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.483492 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.483273 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.483492 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.483295 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxwqn\" (UniqueName: \"kubernetes.io/projected/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-kube-api-access-mxwqn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.483492 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.483324 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.483492 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.483408 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.483737 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.483507 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.483737 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.483546 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.484281 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.484259 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.485569 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.485548 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.485772 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.485751 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.486080 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.486065 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.486167 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.486144 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.502252 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.502222 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.502350 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.502275 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxwqn\" (UniqueName: \"kubernetes.io/projected/5fb4ff80-7ad5-4be5-9f2f-868fea0de363-kube-api-access-mxwqn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zhd8v\" (UID: \"5fb4ff80-7ad5-4be5-9f2f-868fea0de363\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.651181 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.651082 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:21.960861 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:21.960820 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v"] Oct 29 23:18:21.966418 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:18:21.966366 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb4ff80_7ad5_4be5_9f2f_868fea0de363.slice/crio-53641df0bb278f669ee1b14fd7631bf5835e79c3a459024ae8c954c2f716d897 WatchSource:0}: Error finding container 53641df0bb278f669ee1b14fd7631bf5835e79c3a459024ae8c954c2f716d897: Status 404 returned error can't find the container with id 53641df0bb278f669ee1b14fd7631bf5835e79c3a459024ae8c954c2f716d897 Oct 29 23:18:22.758853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:22.758814 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" event={"ID":"5fb4ff80-7ad5-4be5-9f2f-868fea0de363","Type":"ContainerStarted","Data":"53641df0bb278f669ee1b14fd7631bf5835e79c3a459024ae8c954c2f716d897"} Oct 29 23:18:24.155666 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:24.155630 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:18:24.155965 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:24.155697 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:18:24.768072 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:24.768033 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" event={"ID":"5fb4ff80-7ad5-4be5-9f2f-868fea0de363","Type":"ContainerStarted","Data":"3e4fd6cbba00deca81e6591f4bb1753eb67329f8ae694a45de7d2eeba2ec801e"} Oct 29 23:18:24.768241 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:24.768150 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:24.817659 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:24.817605 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" podStartSLOduration=1.6311638830000001 podStartE2EDuration="3.817590249s" totalImagesPullingTime="2.186426366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:24.805962544 +0000 UTC m=+274.168967268" watchObservedRunningTime="2026-10-29 23:18:24.817590249 +0000 UTC m=+274.180594971" Oct 29 23:18:25.777877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:18:25.777840 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zhd8v" Oct 29 23:18:45.222414 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:18:45.222328 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:19:15.222339 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:19:15.222298 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:19:34.503534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:34.503490 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-xtrvr"] Oct 29 23:19:34.506866 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:34.506844 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xtrvr" Oct 29 23:19:34.522721 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:34.522694 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-wc52v\"" Oct 29 23:19:34.534115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:34.534086 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 29 23:19:34.534255 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:34.534129 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 29 23:19:34.534255 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:34.534129 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 29 23:19:34.535299 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:34.535275 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-xtrvr"] Oct 29 23:19:34.611076 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:34.611033 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjsqb\" (UniqueName: \"kubernetes.io/projected/69a9fdeb-90d5-4e2f-8655-c7cd6f5c462d-kube-api-access-wjsqb\") pod \"dns-operator-controller-manager-65bfd79598-xtrvr\" (UID: \"69a9fdeb-90d5-4e2f-8655-c7cd6f5c462d\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xtrvr" Oct 29 23:19:34.712180 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:34.712144 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsqb\" (UniqueName: \"kubernetes.io/projected/69a9fdeb-90d5-4e2f-8655-c7cd6f5c462d-kube-api-access-wjsqb\") pod \"dns-operator-controller-manager-65bfd79598-xtrvr\" (UID: \"69a9fdeb-90d5-4e2f-8655-c7cd6f5c462d\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xtrvr" Oct 29 23:19:34.730351 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:34.730322 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjsqb\" (UniqueName: \"kubernetes.io/projected/69a9fdeb-90d5-4e2f-8655-c7cd6f5c462d-kube-api-access-wjsqb\") pod \"dns-operator-controller-manager-65bfd79598-xtrvr\" (UID: \"69a9fdeb-90d5-4e2f-8655-c7cd6f5c462d\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xtrvr" Oct 29 23:19:34.819523 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:34.819422 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xtrvr" Oct 29 23:19:35.118491 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:35.118386 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-xtrvr"] Oct 29 23:19:35.123658 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:19:35.123627 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a9fdeb_90d5_4e2f_8655_c7cd6f5c462d.slice/crio-2818a07ad25a04a12def70fef47d930849fb3dfb8cb5323daaa738ba61fdbc4d WatchSource:0}: Error finding container 2818a07ad25a04a12def70fef47d930849fb3dfb8cb5323daaa738ba61fdbc4d: Status 404 returned error can't find the container with id 2818a07ad25a04a12def70fef47d930849fb3dfb8cb5323daaa738ba61fdbc4d Oct 29 23:19:36.013450 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:36.013406 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xtrvr" event={"ID":"69a9fdeb-90d5-4e2f-8655-c7cd6f5c462d","Type":"ContainerStarted","Data":"2818a07ad25a04a12def70fef47d930849fb3dfb8cb5323daaa738ba61fdbc4d"} Oct 29 23:19:36.784396 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:36.784366 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn"] Oct 29 23:19:36.794115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:36.794090 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn" Oct 29 23:19:36.830791 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:36.830760 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn"] Oct 29 23:19:36.892313 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:36.892282 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-69xpk\"" Oct 29 23:19:36.928533 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:36.928492 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qw5k\" (UniqueName: \"kubernetes.io/projected/44a98780-1841-4e28-8f13-0a23da0c41ad-kube-api-access-7qw5k\") pod \"limitador-operator-controller-manager-856dd584c5-p6qmn\" (UID: \"44a98780-1841-4e28-8f13-0a23da0c41ad\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn" Oct 29 23:19:37.029622 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:37.029587 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qw5k\" (UniqueName: \"kubernetes.io/projected/44a98780-1841-4e28-8f13-0a23da0c41ad-kube-api-access-7qw5k\") pod \"limitador-operator-controller-manager-856dd584c5-p6qmn\" (UID: \"44a98780-1841-4e28-8f13-0a23da0c41ad\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn" Oct 29 23:19:37.051980 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:37.051918 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qw5k\" (UniqueName: \"kubernetes.io/projected/44a98780-1841-4e28-8f13-0a23da0c41ad-kube-api-access-7qw5k\") pod \"limitador-operator-controller-manager-856dd584c5-p6qmn\" (UID: \"44a98780-1841-4e28-8f13-0a23da0c41ad\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn" Oct 29 23:19:37.109700 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:37.109660 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn" Oct 29 23:19:37.419819 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:37.419793 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn"] Oct 29 23:19:37.432528 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:19:37.432489 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a98780_1841_4e28_8f13_0a23da0c41ad.slice/crio-dc6484e73b94277eac20abda6ac1dd76a92655b06ff8f408619c6920a5ea427d WatchSource:0}: Error finding container dc6484e73b94277eac20abda6ac1dd76a92655b06ff8f408619c6920a5ea427d: Status 404 returned error can't find the container with id dc6484e73b94277eac20abda6ac1dd76a92655b06ff8f408619c6920a5ea427d Oct 29 23:19:38.025257 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:38.025222 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn" event={"ID":"44a98780-1841-4e28-8f13-0a23da0c41ad","Type":"ContainerStarted","Data":"dc6484e73b94277eac20abda6ac1dd76a92655b06ff8f408619c6920a5ea427d"} Oct 29 23:19:38.027103 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:38.027074 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xtrvr" event={"ID":"69a9fdeb-90d5-4e2f-8655-c7cd6f5c462d","Type":"ContainerStarted","Data":"3b2347aba93d84e3597a5cc9c248245899739f91e44613a6dc59b9a94a3bbf56"} Oct 29 23:19:38.027231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:38.027205 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xtrvr" Oct 29 23:19:38.065895 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:38.065835 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xtrvr" podStartSLOduration=1.791124424 podStartE2EDuration="4.065817701s" totalImagesPullingTime="2.274693277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:38.064866186 +0000 UTC m=+347.427870910" watchObservedRunningTime="2026-10-29 23:19:38.065817701 +0000 UTC m=+347.428822425" Oct 29 23:19:39.032930 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:39.032830 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn" event={"ID":"44a98780-1841-4e28-8f13-0a23da0c41ad","Type":"ContainerStarted","Data":"3fbedb5b60cd82a8e496e182e092c5b70b78d029d2d7fb0a46302b4ae21d167a"} Oct 29 23:19:39.033124 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:39.033105 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn" Oct 29 23:19:39.070891 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:39.070835 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn" podStartSLOduration=1.776171782 podStartE2EDuration="3.070821422s" totalImagesPullingTime="1.29464964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:39.070338341 +0000 UTC m=+348.433343066" watchObservedRunningTime="2026-10-29 23:19:39.070821422 +0000 UTC m=+348.433826145" Oct 29 23:19:39.781174 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:39.781142 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-q69nr"] Oct 29 23:19:39.784141 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:39.784120 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-q69nr" Oct 29 23:19:39.799278 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:39.799255 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-7ht64\"" Oct 29 23:19:39.812436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:39.812407 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-q69nr"] Oct 29 23:19:39.854966 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:39.854932 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lnk\" (UniqueName: \"kubernetes.io/projected/8d39ed61-cd5e-4610-ba11-365ee048c1b5-kube-api-access-b5lnk\") pod \"authorino-operator-68d6dd76dc-q69nr\" (UID: \"8d39ed61-cd5e-4610-ba11-365ee048c1b5\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-q69nr" Oct 29 23:19:39.955619 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:39.955585 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lnk\" (UniqueName: \"kubernetes.io/projected/8d39ed61-cd5e-4610-ba11-365ee048c1b5-kube-api-access-b5lnk\") pod \"authorino-operator-68d6dd76dc-q69nr\" (UID: \"8d39ed61-cd5e-4610-ba11-365ee048c1b5\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-q69nr" Oct 29 23:19:39.976630 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:39.976604 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lnk\" (UniqueName: \"kubernetes.io/projected/8d39ed61-cd5e-4610-ba11-365ee048c1b5-kube-api-access-b5lnk\") pod \"authorino-operator-68d6dd76dc-q69nr\" (UID: \"8d39ed61-cd5e-4610-ba11-365ee048c1b5\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-q69nr" Oct 29 23:19:40.096406 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:40.096311 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-q69nr" Oct 29 23:19:40.417829 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:40.417744 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-q69nr"] Oct 29 23:19:40.422415 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:19:40.422381 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d39ed61_cd5e_4610_ba11_365ee048c1b5.slice/crio-3e32c78db201ee5bca10adfb4d55303bdfb7872b18c4bef35291621d09f63de7 WatchSource:0}: Error finding container 3e32c78db201ee5bca10adfb4d55303bdfb7872b18c4bef35291621d09f63de7: Status 404 returned error can't find the container with id 3e32c78db201ee5bca10adfb4d55303bdfb7872b18c4bef35291621d09f63de7 Oct 29 23:19:41.041043 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:41.041006 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-q69nr" event={"ID":"8d39ed61-cd5e-4610-ba11-365ee048c1b5","Type":"ContainerStarted","Data":"3e32c78db201ee5bca10adfb4d55303bdfb7872b18c4bef35291621d09f63de7"} Oct 29 23:19:42.046537 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:42.046502 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-q69nr" event={"ID":"8d39ed61-cd5e-4610-ba11-365ee048c1b5","Type":"ContainerStarted","Data":"ae96cb95a2446a3ca2be650c9ef8ad0b5a76e98e7aff0b8d40e51460c28eac46"} Oct 29 23:19:42.046934 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:42.046583 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-q69nr" Oct 29 23:19:42.083870 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:42.083744 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-q69nr" podStartSLOduration=1.680912824 podStartE2EDuration="3.083725167s" totalImagesPullingTime="1.402812343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:42.082826929 +0000 UTC m=+351.445831651" watchObservedRunningTime="2026-10-29 23:19:42.083725167 +0000 UTC m=+351.446729890" Oct 29 23:19:45.223560 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:19:45.223522 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:19:49.035267 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:49.035232 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xtrvr" Oct 29 23:19:50.038087 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:50.038055 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn" Oct 29 23:19:52.649067 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:52.649032 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn"] Oct 29 23:19:52.649523 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:52.649273 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn" podUID="44a98780-1841-4e28-8f13-0a23da0c41ad" containerName="manager" containerID="cri-o://3fbedb5b60cd82a8e496e182e092c5b70b78d029d2d7fb0a46302b4ae21d167a" gracePeriod=2 Oct 29 23:19:52.663414 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:52.663379 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn"] Oct 29 23:19:52.683440 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:52.683400 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-l89qp"] Oct 29 23:19:52.683885 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:52.683867 6507 cpu_manager.go:414] "containerMap: removing container" podUID="44a98780-1841-4e28-8f13-0a23da0c41ad" containerName="manager" Oct 29 23:19:52.683960 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:52.683888 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a98780-1841-4e28-8f13-0a23da0c41ad" containerName="manager" Oct 29 23:19:52.684025 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:52.683996 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a98780-1841-4e28-8f13-0a23da0c41ad" containerName="manager" Oct 29 23:19:52.687807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:52.687787 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-l89qp" Oct 29 23:19:52.700560 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:19:52.700514 6507 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-p6qmn\" is forbidden: User \"system:node:ip-10-0-133-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-100.ec2.internal' and this object" podUID="44a98780-1841-4e28-8f13-0a23da0c41ad" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn" Oct 29 23:19:52.712130 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:52.712096 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-l89qp"] Oct 29 23:19:52.870036 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:52.870000 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czk7x\" (UniqueName: \"kubernetes.io/projected/cec1507f-b47f-4885-a6a5-3fcaf3a4adff-kube-api-access-czk7x\") pod \"limitador-operator-controller-manager-856dd584c5-l89qp\" (UID: \"cec1507f-b47f-4885-a6a5-3fcaf3a4adff\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-l89qp" Oct 29 23:19:52.895749 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:52.895727 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn" Oct 29 23:19:52.971298 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:52.971260 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czk7x\" (UniqueName: \"kubernetes.io/projected/cec1507f-b47f-4885-a6a5-3fcaf3a4adff-kube-api-access-czk7x\") pod \"limitador-operator-controller-manager-856dd584c5-l89qp\" (UID: \"cec1507f-b47f-4885-a6a5-3fcaf3a4adff\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-l89qp" Oct 29 23:19:52.995317 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:52.995284 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czk7x\" (UniqueName: \"kubernetes.io/projected/cec1507f-b47f-4885-a6a5-3fcaf3a4adff-kube-api-access-czk7x\") pod \"limitador-operator-controller-manager-856dd584c5-l89qp\" (UID: \"cec1507f-b47f-4885-a6a5-3fcaf3a4adff\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-l89qp" Oct 29 23:19:53.013146 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:53.013120 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-l89qp" Oct 29 23:19:53.053016 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:53.052987 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-q69nr" Oct 29 23:19:53.072177 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:53.072148 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a98780-1841-4e28-8f13-0a23da0c41ad-kube-api-access-7qw5k\" (UniqueName: \"kubernetes.io/projected/44a98780-1841-4e28-8f13-0a23da0c41ad-kube-api-access-7qw5k\") pod \"44a98780-1841-4e28-8f13-0a23da0c41ad\" (UID: \"44a98780-1841-4e28-8f13-0a23da0c41ad\") " Oct 29 23:19:53.075879 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:53.075855 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a98780-1841-4e28-8f13-0a23da0c41ad-kube-api-access-7qw5k" pod "44a98780-1841-4e28-8f13-0a23da0c41ad" (UID: "44a98780-1841-4e28-8f13-0a23da0c41ad"). InnerVolumeSpecName "kube-api-access-7qw5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:53.088758 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:53.088726 6507 generic.go:356] "Generic (PLEG): container finished" podID="44a98780-1841-4e28-8f13-0a23da0c41ad" containerID="3fbedb5b60cd82a8e496e182e092c5b70b78d029d2d7fb0a46302b4ae21d167a" exitCode=0 Oct 29 23:19:53.088875 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:53.088770 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-p6qmn" Oct 29 23:19:53.088875 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:53.088814 6507 scope.go:122] "RemoveContainer" containerID="3fbedb5b60cd82a8e496e182e092c5b70b78d029d2d7fb0a46302b4ae21d167a" Oct 29 23:19:53.097417 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:53.097401 6507 scope.go:122] "RemoveContainer" containerID="3fbedb5b60cd82a8e496e182e092c5b70b78d029d2d7fb0a46302b4ae21d167a" Oct 29 23:19:53.097721 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:19:53.097703 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fbedb5b60cd82a8e496e182e092c5b70b78d029d2d7fb0a46302b4ae21d167a\": container with ID starting with 3fbedb5b60cd82a8e496e182e092c5b70b78d029d2d7fb0a46302b4ae21d167a not found: ID does not exist" containerID="3fbedb5b60cd82a8e496e182e092c5b70b78d029d2d7fb0a46302b4ae21d167a" Oct 29 23:19:53.097772 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:53.097732 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fbedb5b60cd82a8e496e182e092c5b70b78d029d2d7fb0a46302b4ae21d167a"} err="failed to get container status \"3fbedb5b60cd82a8e496e182e092c5b70b78d029d2d7fb0a46302b4ae21d167a\": rpc error: code = NotFound desc = could not find container \"3fbedb5b60cd82a8e496e182e092c5b70b78d029d2d7fb0a46302b4ae21d167a\": container with ID starting with 3fbedb5b60cd82a8e496e182e092c5b70b78d029d2d7fb0a46302b4ae21d167a not found: ID does not exist" Oct 29 23:19:53.173564 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:53.173536 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qw5k\" (UniqueName: \"kubernetes.io/projected/44a98780-1841-4e28-8f13-0a23da0c41ad-kube-api-access-7qw5k\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:19:53.315963 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:53.315873 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-l89qp"] Oct 29 23:19:53.322523 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:19:53.322488 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec1507f_b47f_4885_a6a5_3fcaf3a4adff.slice/crio-676bf065d4316a5d5dacd5235f34abdaa8d3eabd0d96fb87e031a125973a2d36 WatchSource:0}: Error finding container 676bf065d4316a5d5dacd5235f34abdaa8d3eabd0d96fb87e031a125973a2d36: Status 404 returned error can't find the container with id 676bf065d4316a5d5dacd5235f34abdaa8d3eabd0d96fb87e031a125973a2d36 Oct 29 23:19:54.095000 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:54.094968 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-l89qp" event={"ID":"cec1507f-b47f-4885-a6a5-3fcaf3a4adff","Type":"ContainerStarted","Data":"6c70b3708a02447d3c6da77b59dad32f491e21b04ff22a96071473a1345227ef"} Oct 29 23:19:54.095439 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:54.095018 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-l89qp" event={"ID":"cec1507f-b47f-4885-a6a5-3fcaf3a4adff","Type":"ContainerStarted","Data":"676bf065d4316a5d5dacd5235f34abdaa8d3eabd0d96fb87e031a125973a2d36"} Oct 29 23:19:54.095439 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:54.095116 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-l89qp" Oct 29 23:19:54.145472 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:54.145411 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-l89qp" podStartSLOduration=2.145397967 podStartE2EDuration="2.145397967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:54.144141316 +0000 UTC m=+363.507146039" watchObservedRunningTime="2026-10-29 23:19:54.145397967 +0000 UTC m=+363.508402690" Oct 29 23:19:54.751319 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:19:54.751283 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a98780-1841-4e28-8f13-0a23da0c41ad" path="/var/lib/kubelet/pods/44a98780-1841-4e28-8f13-0a23da0c41ad/volumes" Oct 29 23:20:05.102211 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:05.102136 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-l89qp" Oct 29 23:20:15.222043 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:20:15.221998 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:20:23.386648 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.386608 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf"] Oct 29 23:20:23.391393 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.391366 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.407892 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.407861 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tvvvz\"" Oct 29 23:20:23.421006 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.420964 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf"] Oct 29 23:20:23.531867 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.531809 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.532042 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.531895 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.532042 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.531921 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.532042 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.531961 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.532042 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.532014 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.532208 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.532045 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvp6c\" (UniqueName: \"kubernetes.io/projected/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-kube-api-access-jvp6c\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.532208 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.532123 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.532208 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.532172 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.532309 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.532220 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.632998 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.632959 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.633175 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.633006 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.633175 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.633028 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.633175 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.633056 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.633175 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.633086 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.633175 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.633109 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvp6c\" (UniqueName: \"kubernetes.io/projected/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-kube-api-access-jvp6c\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.633175 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.633134 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.633175 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.633162 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.633592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.633189 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.633592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.633417 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.633592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.633559 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.633719 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.633614 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.633819 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.633801 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.633993 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.633977 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.635843 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.635820 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.635948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.635855 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.654914 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.654849 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvp6c\" (UniqueName: \"kubernetes.io/projected/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-kube-api-access-jvp6c\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.656299 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.656279 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7084c3a6-6454-456d-a6ad-06f58bf3e4e0-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-bsqhf\" (UID: \"7084c3a6-6454-456d-a6ad-06f58bf3e4e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:23.705215 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:23.705181 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:24.046933 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:24.046900 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf"] Oct 29 23:20:24.050408 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:20:24.050386 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7084c3a6_6454_456d_a6ad_06f58bf3e4e0.slice/crio-2eff7667d8cbdda83352e6b37c17394ecbc7f153c1e55cdd326853f622f6797a WatchSource:0}: Error finding container 2eff7667d8cbdda83352e6b37c17394ecbc7f153c1e55cdd326853f622f6797a: Status 404 returned error can't find the container with id 2eff7667d8cbdda83352e6b37c17394ecbc7f153c1e55cdd326853f622f6797a Oct 29 23:20:24.052387 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:24.052351 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:20:24.052490 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:24.052425 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:20:24.052490 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:24.052453 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:20:24.215883 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:24.215844 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" event={"ID":"7084c3a6-6454-456d-a6ad-06f58bf3e4e0","Type":"ContainerStarted","Data":"c95f3691c05365518a4dfb7c8fb93e437be73219f7de408b483e2b5b2ec35e95"} Oct 29 23:20:24.216063 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:24.215901 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" event={"ID":"7084c3a6-6454-456d-a6ad-06f58bf3e4e0","Type":"ContainerStarted","Data":"2eff7667d8cbdda83352e6b37c17394ecbc7f153c1e55cdd326853f622f6797a"} Oct 29 23:20:24.258434 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:24.258366 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" podStartSLOduration=1.258351694 podStartE2EDuration="1.258351694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:24.257129566 +0000 UTC m=+393.620134289" watchObservedRunningTime="2026-10-29 23:20:24.258351694 +0000 UTC m=+393.621356414" Oct 29 23:20:24.705630 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:24.705580 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:25.709948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:25.709916 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:26.224711 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:26.224673 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:26.225655 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:26.225635 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:20:38.386038 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:38.386007 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:20:38.389381 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:38.389361 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-5dnd2" Oct 29 23:20:38.418946 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:38.418921 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2q4jt\"" Oct 29 23:20:38.418946 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:38.418923 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 29 23:20:38.419939 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:38.419886 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:20:38.469216 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:38.469176 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24jgx\" (UniqueName: \"kubernetes.io/projected/1aa1f873-3de0-4311-abe5-27cf61c905fe-kube-api-access-24jgx\") pod \"limitador-limitador-7c9b87d574-5dnd2\" (UID: \"1aa1f873-3de0-4311-abe5-27cf61c905fe\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5dnd2" Oct 29 23:20:38.469392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:38.469227 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1aa1f873-3de0-4311-abe5-27cf61c905fe-config-file\") pod \"limitador-limitador-7c9b87d574-5dnd2\" (UID: \"1aa1f873-3de0-4311-abe5-27cf61c905fe\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5dnd2" Oct 29 23:20:38.482769 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:38.482734 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:20:38.570542 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:38.570512 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24jgx\" (UniqueName: \"kubernetes.io/projected/1aa1f873-3de0-4311-abe5-27cf61c905fe-kube-api-access-24jgx\") pod \"limitador-limitador-7c9b87d574-5dnd2\" (UID: \"1aa1f873-3de0-4311-abe5-27cf61c905fe\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5dnd2" Oct 29 23:20:38.570714 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:38.570551 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1aa1f873-3de0-4311-abe5-27cf61c905fe-config-file\") pod \"limitador-limitador-7c9b87d574-5dnd2\" (UID: \"1aa1f873-3de0-4311-abe5-27cf61c905fe\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5dnd2" Oct 29 23:20:38.571202 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:38.571180 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1aa1f873-3de0-4311-abe5-27cf61c905fe-config-file\") pod \"limitador-limitador-7c9b87d574-5dnd2\" (UID: \"1aa1f873-3de0-4311-abe5-27cf61c905fe\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5dnd2" Oct 29 23:20:38.590170 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:38.590138 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24jgx\" (UniqueName: \"kubernetes.io/projected/1aa1f873-3de0-4311-abe5-27cf61c905fe-kube-api-access-24jgx\") pod \"limitador-limitador-7c9b87d574-5dnd2\" (UID: \"1aa1f873-3de0-4311-abe5-27cf61c905fe\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5dnd2" Oct 29 23:20:38.702680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:38.702632 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-5dnd2" Oct 29 23:20:38.998745 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:38.998662 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:20:39.003617 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:20:39.003589 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa1f873_3de0_4311_abe5_27cf61c905fe.slice/crio-7ebedf369af235e569d22305142d69289ee5f8cf9b608fd0ca6f9cf001773838 WatchSource:0}: Error finding container 7ebedf369af235e569d22305142d69289ee5f8cf9b608fd0ca6f9cf001773838: Status 404 returned error can't find the container with id 7ebedf369af235e569d22305142d69289ee5f8cf9b608fd0ca6f9cf001773838 Oct 29 23:20:39.277817 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:39.277726 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-5dnd2" event={"ID":"1aa1f873-3de0-4311-abe5-27cf61c905fe","Type":"ContainerStarted","Data":"7ebedf369af235e569d22305142d69289ee5f8cf9b608fd0ca6f9cf001773838"} Oct 29 23:20:40.283009 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:40.282967 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-5dnd2" event={"ID":"1aa1f873-3de0-4311-abe5-27cf61c905fe","Type":"ContainerStarted","Data":"0308e14dcf856fac1ec0bb7f9a6a87d6580a37116eca5fefa69793dd8fa1b2f6"} Oct 29 23:20:40.283428 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:40.283106 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-5dnd2" Oct 29 23:20:40.319266 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:40.319216 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-5dnd2" podStartSLOduration=1.219852701 podStartE2EDuration="2.319203168s" totalImagesPullingTime="1.099350467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:40.318994565 +0000 UTC m=+409.681999287" watchObservedRunningTime="2026-10-29 23:20:40.319203168 +0000 UTC m=+409.682207891" Oct 29 23:20:45.224058 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:20:45.224015 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:20:51.288710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:20:51.288679 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-5dnd2" Oct 29 23:21:15.222915 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:21:15.222877 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:21:45.223044 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:21:45.222961 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:22:00.531782 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:00.531744 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:22:10.667444 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:10.667412 6507 generic.go:356] "Generic (PLEG): container finished" podID="7084c3a6-6454-456d-a6ad-06f58bf3e4e0" containerID="c95f3691c05365518a4dfb7c8fb93e437be73219f7de408b483e2b5b2ec35e95" exitCode=0 Oct 29 23:22:10.668007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:10.667491 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" event={"ID":"7084c3a6-6454-456d-a6ad-06f58bf3e4e0","Type":"ContainerDied","Data":"c95f3691c05365518a4dfb7c8fb93e437be73219f7de408b483e2b5b2ec35e95"} Oct 29 23:22:10.668007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:10.667950 6507 scope.go:122] "RemoveContainer" containerID="c95f3691c05365518a4dfb7c8fb93e437be73219f7de408b483e2b5b2ec35e95" Oct 29 23:22:10.669081 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:10.669054 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:22:10.669138 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:10.669124 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:22:10.669173 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:10.669159 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:22:11.226603 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:11.226560 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:22:11.673112 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:11.673026 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" event={"ID":"7084c3a6-6454-456d-a6ad-06f58bf3e4e0","Type":"ContainerStarted","Data":"ccb1dd0b6bed52932936528d7689dfbb3f5af4ee2707776068df8d0dc2a806f6"} Oct 29 23:22:11.721484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:11.712834 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:22:11.726735 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:11.726699 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bsqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Oct 29 23:22:11.727038 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:11.726989 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" podUID="7084c3a6-6454-456d-a6ad-06f58bf3e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Oct 29 23:22:12.727487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:12.721581 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bsqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Oct 29 23:22:12.727487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:12.721677 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" podUID="7084c3a6-6454-456d-a6ad-06f58bf3e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Oct 29 23:22:13.724950 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:13.724904 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bsqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Oct 29 23:22:13.725219 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:13.725187 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" podUID="7084c3a6-6454-456d-a6ad-06f58bf3e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Oct 29 23:22:14.713484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:14.711878 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bsqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Oct 29 23:22:14.713484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:14.711943 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" podUID="7084c3a6-6454-456d-a6ad-06f58bf3e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Oct 29 23:22:15.245918 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:22:15.245880 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:22:15.713922 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:15.713887 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bsqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Oct 29 23:22:15.714554 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:15.714522 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" podUID="7084c3a6-6454-456d-a6ad-06f58bf3e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Oct 29 23:22:16.722992 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:16.722953 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:22:16.723595 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:16.723566 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:22:16.724362 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:22:16.724334 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" Oct 29 23:22:45.222569 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:22:45.222514 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:23:01.217728 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:01.217641 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:23:03.072673 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.072588 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x"] Oct 29 23:23:03.075356 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.075329 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.090836 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.090810 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 29 23:23:03.101429 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.101397 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 29 23:23:03.101429 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.101418 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-p4krx\"" Oct 29 23:23:03.101631 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.101566 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 29 23:23:03.103572 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.103538 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x"] Oct 29 23:23:03.169853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.169812 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64bb8724-5ff7-4382-bec8-a81679b219fb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.169853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.169857 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64bb8724-5ff7-4382-bec8-a81679b219fb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.170086 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.169878 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64bb8724-5ff7-4382-bec8-a81679b219fb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.170086 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.169899 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzgwl\" (UniqueName: \"kubernetes.io/projected/64bb8724-5ff7-4382-bec8-a81679b219fb-kube-api-access-xzgwl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.170086 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.170056 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64bb8724-5ff7-4382-bec8-a81679b219fb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.170198 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.170091 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64bb8724-5ff7-4382-bec8-a81679b219fb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.271442 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.271397 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64bb8724-5ff7-4382-bec8-a81679b219fb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.271442 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.271449 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64bb8724-5ff7-4382-bec8-a81679b219fb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.271699 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.271497 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64bb8724-5ff7-4382-bec8-a81679b219fb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.271699 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.271531 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzgwl\" (UniqueName: \"kubernetes.io/projected/64bb8724-5ff7-4382-bec8-a81679b219fb-kube-api-access-xzgwl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.271699 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.271591 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64bb8724-5ff7-4382-bec8-a81679b219fb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.271699 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.271632 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64bb8724-5ff7-4382-bec8-a81679b219fb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.271922 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.271899 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64bb8724-5ff7-4382-bec8-a81679b219fb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.271985 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.271925 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64bb8724-5ff7-4382-bec8-a81679b219fb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.272041 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.271982 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64bb8724-5ff7-4382-bec8-a81679b219fb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.274025 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.273995 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64bb8724-5ff7-4382-bec8-a81679b219fb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.274273 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.274256 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64bb8724-5ff7-4382-bec8-a81679b219fb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.290767 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.290734 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzgwl\" (UniqueName: \"kubernetes.io/projected/64bb8724-5ff7-4382-bec8-a81679b219fb-kube-api-access-xzgwl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x\" (UID: \"64bb8724-5ff7-4382-bec8-a81679b219fb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.389043 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.388945 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:03.687276 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.687246 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x"] Oct 29 23:23:03.693344 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:23:03.693300 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bb8724_5ff7_4382_bec8_a81679b219fb.slice/crio-98632e6f3c9a6ade96a52afc95044042e2a369e915e8a1363ff65b4c2aa1a9ce WatchSource:0}: Error finding container 98632e6f3c9a6ade96a52afc95044042e2a369e915e8a1363ff65b4c2aa1a9ce: Status 404 returned error can't find the container with id 98632e6f3c9a6ade96a52afc95044042e2a369e915e8a1363ff65b4c2aa1a9ce Oct 29 23:23:03.695381 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.695356 6507 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:23:03.880550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:03.880511 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" event={"ID":"64bb8724-5ff7-4382-bec8-a81679b219fb","Type":"ContainerStarted","Data":"98632e6f3c9a6ade96a52afc95044042e2a369e915e8a1363ff65b4c2aa1a9ce"} Oct 29 23:23:04.886763 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:04.886711 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" event={"ID":"64bb8724-5ff7-4382-bec8-a81679b219fb","Type":"ContainerStarted","Data":"8d39e3b8c24c3290f22e5c41324bc7b2d1fcf4b207ca6fb75b28113a4f09c9ad"} Oct 29 23:23:04.925979 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:04.925929 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" podStartSLOduration=0.880952443 podStartE2EDuration="1.925914243s" totalImagesPullingTime="1.0449618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:04.925788379 +0000 UTC m=+554.288793112" watchObservedRunningTime="2026-10-29 23:23:04.925914243 +0000 UTC m=+554.288918962" Oct 29 23:23:05.389658 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:05.389619 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:05.700468 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:05.700423 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:23:07.296563 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.296527 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv"] Oct 29 23:23:07.300116 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.300094 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.319276 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.319246 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 29 23:23:07.328683 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.328653 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv"] Oct 29 23:23:07.408924 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.408870 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1020c327-0d07-4228-8124-ce2b9d0413ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.409103 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.409011 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1020c327-0d07-4228-8124-ce2b9d0413ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.409103 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.409041 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1020c327-0d07-4228-8124-ce2b9d0413ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.409103 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.409064 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1020c327-0d07-4228-8124-ce2b9d0413ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.409103 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.409082 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8745m\" (UniqueName: \"kubernetes.io/projected/1020c327-0d07-4228-8124-ce2b9d0413ba-kube-api-access-8745m\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.409240 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.409182 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1020c327-0d07-4228-8124-ce2b9d0413ba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.510348 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.510312 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1020c327-0d07-4228-8124-ce2b9d0413ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.510348 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.510349 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1020c327-0d07-4228-8124-ce2b9d0413ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.510646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.510373 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1020c327-0d07-4228-8124-ce2b9d0413ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.510646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.510391 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8745m\" (UniqueName: \"kubernetes.io/projected/1020c327-0d07-4228-8124-ce2b9d0413ba-kube-api-access-8745m\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.510646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.510426 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1020c327-0d07-4228-8124-ce2b9d0413ba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.510646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.510444 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1020c327-0d07-4228-8124-ce2b9d0413ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.510887 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.510842 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1020c327-0d07-4228-8124-ce2b9d0413ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.510924 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.510905 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1020c327-0d07-4228-8124-ce2b9d0413ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.510957 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.510932 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1020c327-0d07-4228-8124-ce2b9d0413ba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.512722 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.512704 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1020c327-0d07-4228-8124-ce2b9d0413ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.512968 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.512953 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1020c327-0d07-4228-8124-ce2b9d0413ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.528687 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.528657 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8745m\" (UniqueName: \"kubernetes.io/projected/1020c327-0d07-4228-8124-ce2b9d0413ba-kube-api-access-8745m\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv\" (UID: \"1020c327-0d07-4228-8124-ce2b9d0413ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.616904 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.616806 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:07.909483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:07.909386 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv"] Oct 29 23:23:07.913754 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:23:07.913725 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1020c327_0d07_4228_8124_ce2b9d0413ba.slice/crio-a60363c36292fc6db8c91962e183f6dda52b0616b23d211879ba047b1aba3488 WatchSource:0}: Error finding container a60363c36292fc6db8c91962e183f6dda52b0616b23d211879ba047b1aba3488: Status 404 returned error can't find the container with id a60363c36292fc6db8c91962e183f6dda52b0616b23d211879ba047b1aba3488 Oct 29 23:23:08.907846 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:08.907791 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" event={"ID":"1020c327-0d07-4228-8124-ce2b9d0413ba","Type":"ContainerStarted","Data":"dce22e48cab389b2eb289b2ee796dd8e40d21e49a3bc5e218145f38a1ac7d823"} Oct 29 23:23:08.908347 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:08.907864 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" event={"ID":"1020c327-0d07-4228-8124-ce2b9d0413ba","Type":"ContainerStarted","Data":"a60363c36292fc6db8c91962e183f6dda52b0616b23d211879ba047b1aba3488"} Oct 29 23:23:08.947294 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:08.947238 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podStartSLOduration=1.732286281 podStartE2EDuration="1.947221933s" totalImagesPullingTime="214.935652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:08.946721169 +0000 UTC m=+558.309725893" watchObservedRunningTime="2026-10-29 23:23:08.947221933 +0000 UTC m=+558.310226656" Oct 29 23:23:09.617856 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:09.617815 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:11.598992 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.598961 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww"] Oct 29 23:23:11.602798 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.602770 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.618299 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.618276 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 29 23:23:11.631212 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.631182 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww"] Oct 29 23:23:11.651046 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.651018 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8489303-190c-4dcf-8deb-125ecb21210c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.651186 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.651067 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8489303-190c-4dcf-8deb-125ecb21210c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.651186 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.651090 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8489303-190c-4dcf-8deb-125ecb21210c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.651262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.651211 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8489303-190c-4dcf-8deb-125ecb21210c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.651262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.651247 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th2qw\" (UniqueName: \"kubernetes.io/projected/d8489303-190c-4dcf-8deb-125ecb21210c-kube-api-access-th2qw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.651324 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.651313 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8489303-190c-4dcf-8deb-125ecb21210c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.713636 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.713600 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:23:11.752642 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.752608 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8489303-190c-4dcf-8deb-125ecb21210c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.752813 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.752660 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8489303-190c-4dcf-8deb-125ecb21210c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.752872 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.752817 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8489303-190c-4dcf-8deb-125ecb21210c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.752928 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.752869 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8489303-190c-4dcf-8deb-125ecb21210c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.752928 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.752918 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8489303-190c-4dcf-8deb-125ecb21210c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.753035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.752949 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th2qw\" (UniqueName: \"kubernetes.io/projected/d8489303-190c-4dcf-8deb-125ecb21210c-kube-api-access-th2qw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.753110 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.753085 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8489303-190c-4dcf-8deb-125ecb21210c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.753241 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.753218 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8489303-190c-4dcf-8deb-125ecb21210c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.753731 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.753706 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8489303-190c-4dcf-8deb-125ecb21210c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.755166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.755144 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8489303-190c-4dcf-8deb-125ecb21210c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.755479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.755438 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8489303-190c-4dcf-8deb-125ecb21210c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.777399 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.777361 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th2qw\" (UniqueName: \"kubernetes.io/projected/d8489303-190c-4dcf-8deb-125ecb21210c-kube-api-access-th2qw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z7xww\" (UID: \"d8489303-190c-4dcf-8deb-125ecb21210c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:11.917126 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:11.917034 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:12.215171 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:12.215138 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww"] Oct 29 23:23:12.219894 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:23:12.219868 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8489303_190c_4dcf_8deb_125ecb21210c.slice/crio-6ff3ffaf9c1888bb82030e9a85488f7798bfae4209b83f10f4687a8d897305b5 WatchSource:0}: Error finding container 6ff3ffaf9c1888bb82030e9a85488f7798bfae4209b83f10f4687a8d897305b5: Status 404 returned error can't find the container with id 6ff3ffaf9c1888bb82030e9a85488f7798bfae4209b83f10f4687a8d897305b5 Oct 29 23:23:12.927797 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:12.927701 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" event={"ID":"d8489303-190c-4dcf-8deb-125ecb21210c","Type":"ContainerStarted","Data":"29b5217530b153bd89b9a242fd869106f6ce453dac3bde50e5d3899173576a9a"} Oct 29 23:23:12.927797 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:12.927754 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" event={"ID":"d8489303-190c-4dcf-8deb-125ecb21210c","Type":"ContainerStarted","Data":"6ff3ffaf9c1888bb82030e9a85488f7798bfae4209b83f10f4687a8d897305b5"} Oct 29 23:23:12.977005 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:12.976958 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" podStartSLOduration=1.757847923 podStartE2EDuration="1.976945058s" totalImagesPullingTime="219.097135ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:12.9761232 +0000 UTC m=+562.339127922" watchObservedRunningTime="2026-10-29 23:23:12.976945058 +0000 UTC m=+562.339949777" Oct 29 23:23:13.389334 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:13.389297 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:13.402399 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:13.401996 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:13.917917 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:13.917882 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:13.943037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:13.943004 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x" Oct 29 23:23:15.223830 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:23:15.223793 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:23:17.617379 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:17.617338 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:17.629921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:17.629892 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:17.875698 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:17.875611 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:23:17.963470 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:17.963430 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:23:21.918263 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:21.918223 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:21.931149 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:21.931116 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:21.985381 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:21.985352 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" Oct 29 23:23:25.795509 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:25.795452 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx"] Oct 29 23:23:25.799228 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:25.799205 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" Oct 29 23:23:25.820783 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:25.820748 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 29 23:23:25.820783 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:25.820748 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 29 23:23:25.820783 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:25.820752 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-t9dqk\"" Oct 29 23:23:25.822072 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:25.822045 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx"] Oct 29 23:23:25.890085 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:25.890051 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04824e2a-3f12-4065-9c4f-5079ced25842-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-blnsx\" (UID: \"04824e2a-3f12-4065-9c4f-5079ced25842\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" Oct 29 23:23:25.890268 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:25.890107 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tvfh\" (UniqueName: \"kubernetes.io/projected/04824e2a-3f12-4065-9c4f-5079ced25842-kube-api-access-7tvfh\") pod \"ai-gateway-controller-7cb8f6bb64-blnsx\" (UID: \"04824e2a-3f12-4065-9c4f-5079ced25842\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" Oct 29 23:23:25.991064 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:25.991027 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04824e2a-3f12-4065-9c4f-5079ced25842-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-blnsx\" (UID: \"04824e2a-3f12-4065-9c4f-5079ced25842\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" Oct 29 23:23:25.991251 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:25.991079 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tvfh\" (UniqueName: \"kubernetes.io/projected/04824e2a-3f12-4065-9c4f-5079ced25842-kube-api-access-7tvfh\") pod \"ai-gateway-controller-7cb8f6bb64-blnsx\" (UID: \"04824e2a-3f12-4065-9c4f-5079ced25842\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" Oct 29 23:23:25.993623 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:25.993598 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04824e2a-3f12-4065-9c4f-5079ced25842-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-blnsx\" (UID: \"04824e2a-3f12-4065-9c4f-5079ced25842\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" Oct 29 23:23:26.007714 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:26.007681 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tvfh\" (UniqueName: \"kubernetes.io/projected/04824e2a-3f12-4065-9c4f-5079ced25842-kube-api-access-7tvfh\") pod \"ai-gateway-controller-7cb8f6bb64-blnsx\" (UID: \"04824e2a-3f12-4065-9c4f-5079ced25842\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" Oct 29 23:23:26.115529 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:26.115421 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" Oct 29 23:23:26.409206 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:26.409113 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx"] Oct 29 23:23:26.413772 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:23:26.413742 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04824e2a_3f12_4065_9c4f_5079ced25842.slice/crio-f56b703d06c64a61d6a572860f6ff98e1b763ffebc9821a5cd07048f33d3d8f5 WatchSource:0}: Error finding container f56b703d06c64a61d6a572860f6ff98e1b763ffebc9821a5cd07048f33d3d8f5: Status 404 returned error can't find the container with id f56b703d06c64a61d6a572860f6ff98e1b763ffebc9821a5cd07048f33d3d8f5 Oct 29 23:23:27.002479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:27.001978 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" event={"ID":"04824e2a-3f12-4065-9c4f-5079ced25842","Type":"ContainerStarted","Data":"f56b703d06c64a61d6a572860f6ff98e1b763ffebc9821a5cd07048f33d3d8f5"} Oct 29 23:23:29.015041 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:29.014949 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" event={"ID":"04824e2a-3f12-4065-9c4f-5079ced25842","Type":"ContainerStarted","Data":"dd0f22747854f4052cb445ab28e1e69f0e8f42a2ee5442ea769aa4058d5011fc"} Oct 29 23:23:29.015524 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:29.015041 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" Oct 29 23:23:29.048255 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:29.048195 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" podStartSLOduration=1.7422370919999999 podStartE2EDuration="4.048174194s" totalImagesPullingTime="2.305937102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:29.047733961 +0000 UTC m=+578.410738684" watchObservedRunningTime="2026-10-29 23:23:29.048174194 +0000 UTC m=+578.411178916" Oct 29 23:23:39.059230 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.059198 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4"] Oct 29 23:23:39.064343 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.064316 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.090314 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.090281 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4"] Oct 29 23:23:39.116379 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.116338 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.116581 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.116448 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.116581 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.116518 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.116581 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.116548 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.116737 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.116603 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.116737 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.116624 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c45c4\" (UniqueName: \"kubernetes.io/projected/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-kube-api-access-c45c4\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.116737 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.116717 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.116868 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.116738 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.116868 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.116753 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.217705 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.217665 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.217705 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.217706 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.217964 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.217724 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c45c4\" (UniqueName: \"kubernetes.io/projected/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-kube-api-access-c45c4\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.217964 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.217861 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.217964 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.217900 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.217964 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.217922 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.217964 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.217956 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.218254 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.218043 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.218254 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.218081 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.218254 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.218180 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.218254 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.218203 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.218502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.218298 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.218502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.218372 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.218815 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.218795 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.220275 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.220252 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.220576 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.220557 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.234761 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.234726 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.234924 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.234863 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c45c4\" (UniqueName: \"kubernetes.io/projected/cc74a2a4-ee89-46f7-84e9-6e6744b31fb6-kube-api-access-c45c4\") pod \"maas-default-gateway-openshift-default-746ff8f49f-zf2k4\" (UID: \"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.377587 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.377489 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:39.688267 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.688237 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4"] Oct 29 23:23:39.693765 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:23:39.693609 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc74a2a4_ee89_46f7_84e9_6e6744b31fb6.slice/crio-6057bb5d54925d72ed355320dabbf36999b440e5aec60232708add8bbe8d151e WatchSource:0}: Error finding container 6057bb5d54925d72ed355320dabbf36999b440e5aec60232708add8bbe8d151e: Status 404 returned error can't find the container with id 6057bb5d54925d72ed355320dabbf36999b440e5aec60232708add8bbe8d151e Oct 29 23:23:39.695819 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.695786 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:23:39.695882 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.695858 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:23:39.695917 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:39.695886 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:23:40.022074 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.021990 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" Oct 29 23:23:40.066235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.066202 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" event={"ID":"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6","Type":"ContainerStarted","Data":"e938ea7170597a009bcdf2eee9152f7043eb6a637b7bf71beb81ee085f78abc6"} Oct 29 23:23:40.066652 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.066254 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" event={"ID":"cc74a2a4-ee89-46f7-84e9-6e6744b31fb6","Type":"ContainerStarted","Data":"6057bb5d54925d72ed355320dabbf36999b440e5aec60232708add8bbe8d151e"} Oct 29 23:23:40.102661 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.102592 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" podStartSLOduration=1.102572641 podStartE2EDuration="1.102572641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:40.100802309 +0000 UTC m=+589.463807051" watchObservedRunningTime="2026-10-29 23:23:40.102572641 +0000 UTC m=+589.465577366" Oct 29 23:23:40.283976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.283890 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-9b4f55d59-srl67"] Oct 29 23:23:40.288333 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.288307 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-9b4f55d59-srl67" Oct 29 23:23:40.316483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.310728 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-9b4f55d59-srl67"] Oct 29 23:23:40.378164 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.378125 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:40.379654 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.379622 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-zf2k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 29 23:23:40.379791 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.379678 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" podUID="cc74a2a4-ee89-46f7-84e9-6e6744b31fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 29 23:23:40.430764 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.430721 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm4kv\" (UniqueName: \"kubernetes.io/projected/ceac094c-fb24-4f92-906c-46d6df96dcda-kube-api-access-cm4kv\") pod \"ai-gateway-controller-9b4f55d59-srl67\" (UID: \"ceac094c-fb24-4f92-906c-46d6df96dcda\") " pod="opendatahub/ai-gateway-controller-9b4f55d59-srl67" Oct 29 23:23:40.430963 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.430817 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceac094c-fb24-4f92-906c-46d6df96dcda-metrics-certs\") pod \"ai-gateway-controller-9b4f55d59-srl67\" (UID: \"ceac094c-fb24-4f92-906c-46d6df96dcda\") " pod="opendatahub/ai-gateway-controller-9b4f55d59-srl67" Oct 29 23:23:40.531426 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.531390 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm4kv\" (UniqueName: \"kubernetes.io/projected/ceac094c-fb24-4f92-906c-46d6df96dcda-kube-api-access-cm4kv\") pod \"ai-gateway-controller-9b4f55d59-srl67\" (UID: \"ceac094c-fb24-4f92-906c-46d6df96dcda\") " pod="opendatahub/ai-gateway-controller-9b4f55d59-srl67" Oct 29 23:23:40.531426 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.531430 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceac094c-fb24-4f92-906c-46d6df96dcda-metrics-certs\") pod \"ai-gateway-controller-9b4f55d59-srl67\" (UID: \"ceac094c-fb24-4f92-906c-46d6df96dcda\") " pod="opendatahub/ai-gateway-controller-9b4f55d59-srl67" Oct 29 23:23:40.534093 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.534024 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceac094c-fb24-4f92-906c-46d6df96dcda-metrics-certs\") pod \"ai-gateway-controller-9b4f55d59-srl67\" (UID: \"ceac094c-fb24-4f92-906c-46d6df96dcda\") " pod="opendatahub/ai-gateway-controller-9b4f55d59-srl67" Oct 29 23:23:40.548171 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.548141 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm4kv\" (UniqueName: \"kubernetes.io/projected/ceac094c-fb24-4f92-906c-46d6df96dcda-kube-api-access-cm4kv\") pod \"ai-gateway-controller-9b4f55d59-srl67\" (UID: \"ceac094c-fb24-4f92-906c-46d6df96dcda\") " pod="opendatahub/ai-gateway-controller-9b4f55d59-srl67" Oct 29 23:23:40.623481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.622651 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-9b4f55d59-srl67" Oct 29 23:23:40.927571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:40.927542 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-9b4f55d59-srl67"] Oct 29 23:23:40.929574 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:23:40.929542 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceac094c_fb24_4f92_906c_46d6df96dcda.slice/crio-769503f5229b5ddd5085a78441c29eb9209e55e70a8ca6d1ded280f4f2d70871 WatchSource:0}: Error finding container 769503f5229b5ddd5085a78441c29eb9209e55e70a8ca6d1ded280f4f2d70871: Status 404 returned error can't find the container with id 769503f5229b5ddd5085a78441c29eb9209e55e70a8ca6d1ded280f4f2d70871 Oct 29 23:23:41.071950 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:41.071912 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-9b4f55d59-srl67" event={"ID":"ceac094c-fb24-4f92-906c-46d6df96dcda","Type":"ContainerStarted","Data":"769503f5229b5ddd5085a78441c29eb9209e55e70a8ca6d1ded280f4f2d70871"} Oct 29 23:23:41.378692 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:41.378663 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-zf2k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 29 23:23:41.378850 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:41.378708 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" podUID="cc74a2a4-ee89-46f7-84e9-6e6744b31fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 29 23:23:42.078219 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:42.078181 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-9b4f55d59-srl67" event={"ID":"ceac094c-fb24-4f92-906c-46d6df96dcda","Type":"ContainerStarted","Data":"c01b93029c79f3cdaa157a10345d3dbf9a31b0cef86cdf262f0972461c17bdf0"} Oct 29 23:23:42.378083 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:42.377989 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-zf2k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 29 23:23:42.378083 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:42.378049 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" podUID="cc74a2a4-ee89-46f7-84e9-6e6744b31fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 29 23:23:43.083547 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:43.083517 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-9b4f55d59-srl67" Oct 29 23:23:43.378365 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:43.378269 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-zf2k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 29 23:23:43.378365 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:43.378335 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" podUID="cc74a2a4-ee89-46f7-84e9-6e6744b31fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 29 23:23:44.378079 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:44.378044 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-zf2k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 29 23:23:44.378488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:44.378101 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" podUID="cc74a2a4-ee89-46f7-84e9-6e6744b31fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 29 23:23:45.247409 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:23:45.247373 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:23:45.378617 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:45.378575 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-zf2k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 29 23:23:45.379031 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:45.378635 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" podUID="cc74a2a4-ee89-46f7-84e9-6e6744b31fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 29 23:23:46.378728 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:46.378693 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-zf2k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 29 23:23:46.379117 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:46.378748 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" podUID="cc74a2a4-ee89-46f7-84e9-6e6744b31fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 29 23:23:48.055362 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:48.055334 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:48.056237 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:48.056210 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:48.056984 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:48.056966 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-zf2k4" Oct 29 23:23:48.103993 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:48.103944 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-9b4f55d59-srl67" podStartSLOduration=7.773062798 podStartE2EDuration="8.10392889s" totalImagesPullingTime="330.866092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:42.12472734 +0000 UTC m=+591.487732065" watchObservedRunningTime="2026-10-29 23:23:48.10392889 +0000 UTC m=+597.466933612" Oct 29 23:23:54.090602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.090572 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-9b4f55d59-srl67" Oct 29 23:23:54.113986 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.113953 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht"] Oct 29 23:23:54.118593 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.118570 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.133083 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.133045 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f116d226-bd4f-41bd-aba4-06958eccb990-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.133083 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.133104 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f116d226-bd4f-41bd-aba4-06958eccb990-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.133420 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.133227 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f116d226-bd4f-41bd-aba4-06958eccb990-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.133420 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.133287 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f116d226-bd4f-41bd-aba4-06958eccb990-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.133420 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.133321 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-958jb\" (UniqueName: \"kubernetes.io/projected/f116d226-bd4f-41bd-aba4-06958eccb990-kube-api-access-958jb\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.133652 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.133435 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f116d226-bd4f-41bd-aba4-06958eccb990-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.133652 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.133523 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f116d226-bd4f-41bd-aba4-06958eccb990-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.133652 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.133643 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f116d226-bd4f-41bd-aba4-06958eccb990-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.133806 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.133661 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f116d226-bd4f-41bd-aba4-06958eccb990-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.139770 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.139747 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht"] Oct 29 23:23:54.156869 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.156835 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx"] Oct 29 23:23:54.157121 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.157090 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" podUID="04824e2a-3f12-4065-9c4f-5079ced25842" containerName="manager" containerID="cri-o://dd0f22747854f4052cb445ab28e1e69f0e8f42a2ee5442ea769aa4058d5011fc" gracePeriod=30 Oct 29 23:23:54.234607 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.234566 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f116d226-bd4f-41bd-aba4-06958eccb990-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.234607 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.234610 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f116d226-bd4f-41bd-aba4-06958eccb990-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.234830 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.234627 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-958jb\" (UniqueName: \"kubernetes.io/projected/f116d226-bd4f-41bd-aba4-06958eccb990-kube-api-access-958jb\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.234830 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.234743 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f116d226-bd4f-41bd-aba4-06958eccb990-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.234830 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.234795 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f116d226-bd4f-41bd-aba4-06958eccb990-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.235009 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.234849 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f116d226-bd4f-41bd-aba4-06958eccb990-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.235009 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.234872 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f116d226-bd4f-41bd-aba4-06958eccb990-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.235009 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.234943 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f116d226-bd4f-41bd-aba4-06958eccb990-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.235161 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.235016 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f116d226-bd4f-41bd-aba4-06958eccb990-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.235253 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.235224 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f116d226-bd4f-41bd-aba4-06958eccb990-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.235344 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.235321 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f116d226-bd4f-41bd-aba4-06958eccb990-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.235405 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.235374 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f116d226-bd4f-41bd-aba4-06958eccb990-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.235405 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.235375 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f116d226-bd4f-41bd-aba4-06958eccb990-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.235533 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.235440 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f116d226-bd4f-41bd-aba4-06958eccb990-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.237554 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.237528 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f116d226-bd4f-41bd-aba4-06958eccb990-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.237708 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.237690 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f116d226-bd4f-41bd-aba4-06958eccb990-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.251531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.251498 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f116d226-bd4f-41bd-aba4-06958eccb990-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.251664 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.251615 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-958jb\" (UniqueName: \"kubernetes.io/projected/f116d226-bd4f-41bd-aba4-06958eccb990-kube-api-access-958jb\") pod \"maas-default-gateway-openshift-default-746ff8f49f-9phht\" (UID: \"f116d226-bd4f-41bd-aba4-06958eccb990\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.394020 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.393994 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" Oct 29 23:23:54.434950 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.434913 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:54.437591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.437549 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04824e2a-3f12-4065-9c4f-5079ced25842-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04824e2a-3f12-4065-9c4f-5079ced25842-metrics-certs\") pod \"04824e2a-3f12-4065-9c4f-5079ced25842\" (UID: \"04824e2a-3f12-4065-9c4f-5079ced25842\") " Oct 29 23:23:54.437778 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.437674 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04824e2a-3f12-4065-9c4f-5079ced25842-kube-api-access-7tvfh\" (UniqueName: \"kubernetes.io/projected/04824e2a-3f12-4065-9c4f-5079ced25842-kube-api-access-7tvfh\") pod \"04824e2a-3f12-4065-9c4f-5079ced25842\" (UID: \"04824e2a-3f12-4065-9c4f-5079ced25842\") " Oct 29 23:23:54.440092 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.440060 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04824e2a-3f12-4065-9c4f-5079ced25842-metrics-certs" pod "04824e2a-3f12-4065-9c4f-5079ced25842" (UID: "04824e2a-3f12-4065-9c4f-5079ced25842"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:23:54.440092 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.440075 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04824e2a-3f12-4065-9c4f-5079ced25842-kube-api-access-7tvfh" pod "04824e2a-3f12-4065-9c4f-5079ced25842" (UID: "04824e2a-3f12-4065-9c4f-5079ced25842"). InnerVolumeSpecName "kube-api-access-7tvfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:54.539400 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.539327 6507 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04824e2a-3f12-4065-9c4f-5079ced25842-metrics-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:23:54.539400 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.539371 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tvfh\" (UniqueName: \"kubernetes.io/projected/04824e2a-3f12-4065-9c4f-5079ced25842-kube-api-access-7tvfh\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:23:54.743311 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.743226 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht"] Oct 29 23:23:54.747887 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:23:54.747860 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf116d226_bd4f_41bd_aba4_06958eccb990.slice/crio-c0aa7b0c4bd8e19a5ced38581bba5673a1fa558ddc50146b6a5da526204f3def WatchSource:0}: Error finding container c0aa7b0c4bd8e19a5ced38581bba5673a1fa558ddc50146b6a5da526204f3def: Status 404 returned error can't find the container with id c0aa7b0c4bd8e19a5ced38581bba5673a1fa558ddc50146b6a5da526204f3def Oct 29 23:23:54.750405 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.750374 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:23:54.750511 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.750440 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:23:54.750511 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:54.750487 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:23:55.137647 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:55.137558 6507 generic.go:356] "Generic (PLEG): container finished" podID="04824e2a-3f12-4065-9c4f-5079ced25842" containerID="dd0f22747854f4052cb445ab28e1e69f0e8f42a2ee5442ea769aa4058d5011fc" exitCode=0 Oct 29 23:23:55.138091 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:55.137644 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" Oct 29 23:23:55.138091 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:55.137652 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" event={"ID":"04824e2a-3f12-4065-9c4f-5079ced25842","Type":"ContainerDied","Data":"dd0f22747854f4052cb445ab28e1e69f0e8f42a2ee5442ea769aa4058d5011fc"} Oct 29 23:23:55.138091 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:55.137710 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx" event={"ID":"04824e2a-3f12-4065-9c4f-5079ced25842","Type":"ContainerDied","Data":"f56b703d06c64a61d6a572860f6ff98e1b763ffebc9821a5cd07048f33d3d8f5"} Oct 29 23:23:55.138091 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:55.137742 6507 scope.go:122] "RemoveContainer" containerID="dd0f22747854f4052cb445ab28e1e69f0e8f42a2ee5442ea769aa4058d5011fc" Oct 29 23:23:55.139477 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:55.139429 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" event={"ID":"f116d226-bd4f-41bd-aba4-06958eccb990","Type":"ContainerStarted","Data":"276d44880ce8bc4d0e1eb211d954e0a2fe64bd15cc0320fce928d9f567e89257"} Oct 29 23:23:55.139599 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:55.139497 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" event={"ID":"f116d226-bd4f-41bd-aba4-06958eccb990","Type":"ContainerStarted","Data":"c0aa7b0c4bd8e19a5ced38581bba5673a1fa558ddc50146b6a5da526204f3def"} Oct 29 23:23:55.146508 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:55.146487 6507 scope.go:122] "RemoveContainer" containerID="dd0f22747854f4052cb445ab28e1e69f0e8f42a2ee5442ea769aa4058d5011fc" Oct 29 23:23:55.146809 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:23:55.146787 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd0f22747854f4052cb445ab28e1e69f0e8f42a2ee5442ea769aa4058d5011fc\": container with ID starting with dd0f22747854f4052cb445ab28e1e69f0e8f42a2ee5442ea769aa4058d5011fc not found: ID does not exist" containerID="dd0f22747854f4052cb445ab28e1e69f0e8f42a2ee5442ea769aa4058d5011fc" Oct 29 23:23:55.146918 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:55.146814 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd0f22747854f4052cb445ab28e1e69f0e8f42a2ee5442ea769aa4058d5011fc"} err="failed to get container status \"dd0f22747854f4052cb445ab28e1e69f0e8f42a2ee5442ea769aa4058d5011fc\": rpc error: code = NotFound desc = could not find container \"dd0f22747854f4052cb445ab28e1e69f0e8f42a2ee5442ea769aa4058d5011fc\": container with ID starting with dd0f22747854f4052cb445ab28e1e69f0e8f42a2ee5442ea769aa4058d5011fc not found: ID does not exist" Oct 29 23:23:55.182835 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:55.182789 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx"] Oct 29 23:23:55.185248 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:55.185217 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-blnsx"] Oct 29 23:23:55.435247 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:55.435208 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:23:55.436770 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:55.436742 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-9phht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 29 23:23:55.436875 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:55.436792 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" podUID="f116d226-bd4f-41bd-aba4-06958eccb990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 29 23:23:56.436204 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:56.436164 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-9phht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 29 23:23:56.436698 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:56.436222 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" podUID="f116d226-bd4f-41bd-aba4-06958eccb990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 29 23:23:56.751512 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:56.751404 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04824e2a-3f12-4065-9c4f-5079ced25842" path="/var/lib/kubelet/pods/04824e2a-3f12-4065-9c4f-5079ced25842/volumes" Oct 29 23:23:57.436151 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:57.436117 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-9phht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 29 23:23:57.436340 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:57.436167 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" podUID="f116d226-bd4f-41bd-aba4-06958eccb990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 29 23:23:58.435763 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:58.435727 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-9phht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 29 23:23:58.435923 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:58.435782 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" podUID="f116d226-bd4f-41bd-aba4-06958eccb990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 29 23:23:59.435604 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:59.435564 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-9phht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 29 23:23:59.436114 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:23:59.435625 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" podUID="f116d226-bd4f-41bd-aba4-06958eccb990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 29 23:24:00.436293 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:00.436255 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-9phht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 29 23:24:00.436736 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:00.436315 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" podUID="f116d226-bd4f-41bd-aba4-06958eccb990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 29 23:24:01.435853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:01.435807 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-9phht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 29 23:24:01.436035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:01.435872 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" podUID="f116d226-bd4f-41bd-aba4-06958eccb990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 29 23:24:02.978220 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:02.978188 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:24:02.978722 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:02.978642 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:24:02.979618 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:02.979568 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" Oct 29 23:24:03.032662 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:03.032608 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" podStartSLOduration=9.032594953 podStartE2EDuration="9.032594953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:55.22238061 +0000 UTC m=+604.585385337" watchObservedRunningTime="2026-10-29 23:24:03.032594953 +0000 UTC m=+612.395599668" Oct 29 23:24:15.223133 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:24:15.223093 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:24:23.520153 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.520110 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps"] Oct 29 23:24:23.520726 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.520706 6507 cpu_manager.go:414] "containerMap: removing container" podUID="04824e2a-3f12-4065-9c4f-5079ced25842" containerName="manager" Oct 29 23:24:23.520793 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.520729 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04824e2a-3f12-4065-9c4f-5079ced25842" containerName="manager" Oct 29 23:24:23.520844 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.520811 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="04824e2a-3f12-4065-9c4f-5079ced25842" containerName="manager" Oct 29 23:24:23.524762 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.524734 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.537772 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.537744 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-be8726-openshift-default-dockercfg-mgctq\"" Oct 29 23:24:23.548202 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.548159 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps"] Oct 29 23:24:23.610761 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.610720 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-credential-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.610959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.610775 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-workload-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.610959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.610832 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-workload-certs\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.610959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.610916 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-envoy\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.611126 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.610955 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-data\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.611126 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.610987 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-podinfo\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.611126 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.611013 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjh9q\" (UniqueName: \"kubernetes.io/projected/6bf4f192-0296-4a17-abf7-b6a55b12f60c-kube-api-access-qjh9q\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.611277 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.611142 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-token\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.611277 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.611160 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istiod-ca-cert\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.712316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.712280 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-token\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.712316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.712316 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istiod-ca-cert\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.712604 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.712352 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-credential-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.712604 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.712384 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-workload-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.712604 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.712400 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-workload-certs\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.712604 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.712420 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-envoy\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.712604 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.712445 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-data\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.712604 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.712484 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-podinfo\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.712604 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.712507 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjh9q\" (UniqueName: \"kubernetes.io/projected/6bf4f192-0296-4a17-abf7-b6a55b12f60c-kube-api-access-qjh9q\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.713261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.712882 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-credential-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.713261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.712950 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-workload-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.713261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.713205 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-workload-certs\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.713389 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.713357 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-data\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.713602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.713578 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istiod-ca-cert\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.715036 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.715014 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-envoy\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.715321 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.715300 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-podinfo\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.730327 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.730290 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-token\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.730489 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.730471 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjh9q\" (UniqueName: \"kubernetes.io/projected/6bf4f192-0296-4a17-abf7-b6a55b12f60c-kube-api-access-qjh9q\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.799868 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.799767 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc"] Oct 29 23:24:23.805011 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.804981 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:23.838516 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.838443 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc"] Oct 29 23:24:23.840047 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.840006 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:23.879003 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.878881 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9280e4-openshift-default-dockercfg-2jl2d\"" Oct 29 23:24:23.917496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.915039 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-token\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:23.917496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.915078 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istiod-ca-cert\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:23.917496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.915103 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-envoy\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:23.917496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.915135 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-workload-socket\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:23.917496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.915153 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-credential-socket\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:23.917496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.915169 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-workload-certs\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:23.917496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.915194 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-data\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:23.917496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.915211 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-podinfo\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:23.917496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:23.915238 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfjmk\" (UniqueName: \"kubernetes.io/projected/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-kube-api-access-hfjmk\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.016755 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.016717 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-data\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.016755 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.016758 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-podinfo\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.016976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.016804 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfjmk\" (UniqueName: \"kubernetes.io/projected/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-kube-api-access-hfjmk\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.016976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.016868 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-token\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.016976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.016892 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istiod-ca-cert\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.016976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.016919 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-envoy\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.016976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.016963 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-workload-socket\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.017225 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.016991 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-credential-socket\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.017225 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.017015 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-workload-certs\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.017225 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.017163 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-data\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.017380 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.017287 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-workload-socket\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.017471 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.017437 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-credential-socket\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.017535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.017516 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-workload-certs\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.017657 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.017641 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istiod-ca-cert\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.019247 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.019223 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-envoy\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.019487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.019441 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-podinfo\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.047630 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.047592 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-token\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.048036 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.048015 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfjmk\" (UniqueName: \"kubernetes.io/projected/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-kube-api-access-hfjmk\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.123897 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.123806 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:24.175785 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.174995 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps"] Oct 29 23:24:24.182053 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:24:24.181870 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf4f192_0296_4a17_abf7_b6a55b12f60c.slice/crio-ce047f8142524409485c03919e25e1007452abd0ae47417f0f9f6c8a3aac7969 WatchSource:0}: Error finding container ce047f8142524409485c03919e25e1007452abd0ae47417f0f9f6c8a3aac7969: Status 404 returned error can't find the container with id ce047f8142524409485c03919e25e1007452abd0ae47417f0f9f6c8a3aac7969 Oct 29 23:24:24.184362 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.184323 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:24.184483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.184415 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:24.184483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.184473 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:24.276168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.276062 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" event={"ID":"6bf4f192-0296-4a17-abf7-b6a55b12f60c","Type":"ContainerStarted","Data":"21ac6a31e98cbcf718f5c10e510b40b8fda554c41f516d4b6842a0a56a583170"} Oct 29 23:24:24.276168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.276126 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" event={"ID":"6bf4f192-0296-4a17-abf7-b6a55b12f60c","Type":"ContainerStarted","Data":"ce047f8142524409485c03919e25e1007452abd0ae47417f0f9f6c8a3aac7969"} Oct 29 23:24:24.314984 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.314739 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" podStartSLOduration=1.3147073470000001 podStartE2EDuration="1.314707347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:24.31411756 +0000 UTC m=+633.677122284" watchObservedRunningTime="2026-10-29 23:24:24.314707347 +0000 UTC m=+633.677712070" Oct 29 23:24:24.332014 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.331868 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4"] Oct 29 23:24:24.337494 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.336338 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.350750 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.350723 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-837245-openshift-default-dockercfg-tn2ps\"" Oct 29 23:24:24.360958 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.360925 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4"] Oct 29 23:24:24.420734 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.420627 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa280316-fb6c-40db-8658-65b84bb4382b-istio-token\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.420937 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.420756 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc4d9\" (UniqueName: \"kubernetes.io/projected/aa280316-fb6c-40db-8658-65b84bb4382b-kube-api-access-pc4d9\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.420937 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.420826 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-workload-certs\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.420937 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.420918 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa280316-fb6c-40db-8658-65b84bb4382b-istio-podinfo\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.421140 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.421071 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-workload-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.421140 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.421105 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-istio-envoy\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.421259 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.421140 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa280316-fb6c-40db-8658-65b84bb4382b-istiod-ca-cert\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.421259 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.421236 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-istio-data\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.421259 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.421257 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-credential-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.428686 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.428645 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc"] Oct 29 23:24:24.433250 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:24:24.433221 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac751e7_bb4e_4bf7_81f2_26fee5a478f3.slice/crio-dd0292da219233ffc9fc1a6feae2e2a09d6f678bcc5e77fce45d2ed98db9e305 WatchSource:0}: Error finding container dd0292da219233ffc9fc1a6feae2e2a09d6f678bcc5e77fce45d2ed98db9e305: Status 404 returned error can't find the container with id dd0292da219233ffc9fc1a6feae2e2a09d6f678bcc5e77fce45d2ed98db9e305 Oct 29 23:24:24.436107 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.436074 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:24.436213 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.436155 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:24.436213 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.436199 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:24.473808 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.473771 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr"] Oct 29 23:24:24.485881 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.485852 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.498687 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.498655 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2670a5-openshift-default-dockercfg-6sq5q\"" Oct 29 23:24:24.503852 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.503819 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr"] Oct 29 23:24:24.524175 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.524141 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc4d9\" (UniqueName: \"kubernetes.io/projected/aa280316-fb6c-40db-8658-65b84bb4382b-kube-api-access-pc4d9\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.524830 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.524237 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-workload-certs\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.524830 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.524293 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa280316-fb6c-40db-8658-65b84bb4382b-istio-podinfo\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.524830 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.524349 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-workload-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.524830 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.524422 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-istio-envoy\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.524830 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.524472 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa280316-fb6c-40db-8658-65b84bb4382b-istiod-ca-cert\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.524830 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.524508 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-istio-data\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.524830 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.524538 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-credential-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.524830 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.524599 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa280316-fb6c-40db-8658-65b84bb4382b-istio-token\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.525262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.525116 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-workload-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.525355 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.525336 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-workload-certs\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.525548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.525528 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-istio-data\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.525771 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.525752 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-credential-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.525870 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.525853 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa280316-fb6c-40db-8658-65b84bb4382b-istiod-ca-cert\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.527519 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.527496 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-istio-envoy\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.527880 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.527853 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa280316-fb6c-40db-8658-65b84bb4382b-istio-podinfo\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.542242 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.542211 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa280316-fb6c-40db-8658-65b84bb4382b-istio-token\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.542390 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.542334 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc4d9\" (UniqueName: \"kubernetes.io/projected/aa280316-fb6c-40db-8658-65b84bb4382b-kube-api-access-pc4d9\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.628210 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.627945 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-data\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.628210 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.627996 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-token\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.628210 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.628055 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-envoy\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.628210 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.628083 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-workload-certs\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.628210 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.628129 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-workload-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.628635 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.628235 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istiod-ca-cert\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.628635 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.628394 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-credential-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.628635 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.628441 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-podinfo\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.628635 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.628492 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z7gt\" (UniqueName: \"kubernetes.io/projected/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-kube-api-access-8z7gt\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.656761 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.656724 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:24.729703 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.729125 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-podinfo\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.729703 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.729168 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z7gt\" (UniqueName: \"kubernetes.io/projected/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-kube-api-access-8z7gt\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.729703 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.729203 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-data\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.729703 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.729217 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-token\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.729703 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.729250 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-envoy\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.729703 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.729266 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-workload-certs\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.729703 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.729291 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-workload-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.729703 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.729328 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istiod-ca-cert\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.729703 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.729367 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-credential-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.730564 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.730507 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-workload-certs\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.731028 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.730764 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-credential-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.731492 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.731167 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-workload-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.731492 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.731319 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-data\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.731677 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.731652 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istiod-ca-cert\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.739027 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.736345 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-podinfo\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.740283 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.740230 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-envoy\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.748854 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.748538 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z7gt\" (UniqueName: \"kubernetes.io/projected/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-kube-api-access-8z7gt\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.756376 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.756185 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-token\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.809480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.809426 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:24.842761 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.842720 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:24.992948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:24.992911 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4"] Oct 29 23:24:24.999114 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:24:24.999082 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa280316_fb6c_40db_8658_65b84bb4382b.slice/crio-5f098b073283e5725390db5b3a94517fcae982acad7e40a034b50ac43afb445b WatchSource:0}: Error finding container 5f098b073283e5725390db5b3a94517fcae982acad7e40a034b50ac43afb445b: Status 404 returned error can't find the container with id 5f098b073283e5725390db5b3a94517fcae982acad7e40a034b50ac43afb445b Oct 29 23:24:25.001612 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.001575 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:25.001695 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.001646 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:25.001695 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.001674 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:25.115732 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.115687 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr"] Oct 29 23:24:25.119635 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:24:25.119595 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbdb5a2_f2e7_4d1b_946a_8d35cacdb9e2.slice/crio-a41e6cc35cb00e23a9d35dc08d0f3458b975de49ee4e58111ee5e7e44ebcf8ce WatchSource:0}: Error finding container a41e6cc35cb00e23a9d35dc08d0f3458b975de49ee4e58111ee5e7e44ebcf8ce: Status 404 returned error can't find the container with id a41e6cc35cb00e23a9d35dc08d0f3458b975de49ee4e58111ee5e7e44ebcf8ce Oct 29 23:24:25.122398 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.122359 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:25.122501 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.122448 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:25.122555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.122537 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:25.284706 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.283249 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" event={"ID":"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2","Type":"ContainerStarted","Data":"a48503f9e10bea7b5196858f404217cf7b5d7b2bb0c153df2c0030ad1748b171"} Oct 29 23:24:25.284706 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.283318 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" event={"ID":"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2","Type":"ContainerStarted","Data":"a41e6cc35cb00e23a9d35dc08d0f3458b975de49ee4e58111ee5e7e44ebcf8ce"} Oct 29 23:24:25.285088 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.285063 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" event={"ID":"aa280316-fb6c-40db-8658-65b84bb4382b","Type":"ContainerStarted","Data":"369f12f078ada2771f8e0a12bb8823e4ef31052e6dede14826fd54975da29523"} Oct 29 23:24:25.285140 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.285119 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" event={"ID":"aa280316-fb6c-40db-8658-65b84bb4382b","Type":"ContainerStarted","Data":"5f098b073283e5725390db5b3a94517fcae982acad7e40a034b50ac43afb445b"} Oct 29 23:24:25.286805 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.286722 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" event={"ID":"dac751e7-bb4e-4bf7-81f2-26fee5a478f3","Type":"ContainerStarted","Data":"606ae1be70bbb339138254abb97a9ce15e8334b24e3344ff19e42a6c40ff9567"} Oct 29 23:24:25.286805 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.286769 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" event={"ID":"dac751e7-bb4e-4bf7-81f2-26fee5a478f3","Type":"ContainerStarted","Data":"dd0292da219233ffc9fc1a6feae2e2a09d6f678bcc5e77fce45d2ed98db9e305"} Oct 29 23:24:25.321496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.320992 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" podStartSLOduration=1.320974232 podStartE2EDuration="1.320974232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:25.320335447 +0000 UTC m=+634.683340171" watchObservedRunningTime="2026-10-29 23:24:25.320974232 +0000 UTC m=+634.683978955" Oct 29 23:24:25.358001 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.357938 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" podStartSLOduration=1.357916611 podStartE2EDuration="1.357916611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:25.355267663 +0000 UTC m=+634.718272385" watchObservedRunningTime="2026-10-29 23:24:25.357916611 +0000 UTC m=+634.720921337" Oct 29 23:24:25.398703 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.398612 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" podStartSLOduration=2.3985929280000002 podStartE2EDuration="2.398592928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:25.397702529 +0000 UTC m=+634.760707265" watchObservedRunningTime="2026-10-29 23:24:25.398592928 +0000 UTC m=+634.761597652" Oct 29 23:24:25.658329 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.658243 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:25.810600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.810552 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:25.855820 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:25.855779 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:26.124809 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:26.124772 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:26.129891 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:26.129860 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:26.292839 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:26.292793 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:26.293034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:26.292857 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:26.293247 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:26.293226 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:24:26.293329 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:26.293282 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:26.663433 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:26.663399 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:26.814977 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:26.814944 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:27.297430 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:27.297389 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:27.297624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:27.297447 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:27.298214 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:27.298187 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:27.298366 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:27.298342 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:31.555981 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.555896 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z"] Oct 29 23:24:31.560508 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.560486 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" Oct 29 23:24:31.576663 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.576636 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-032fa8\"" Oct 29 23:24:31.584031 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.584003 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-032fa8-dockercfg-glshn\"" Oct 29 23:24:31.584664 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.584648 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z"] Oct 29 23:24:31.599035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.599009 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c831dd-76a8-4e42-90b4-19847ae771e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z\" (UID: \"50c831dd-76a8-4e42-90b4-19847ae771e4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" Oct 29 23:24:31.599148 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.599085 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n596b\" (UniqueName: \"kubernetes.io/projected/50c831dd-76a8-4e42-90b4-19847ae771e4-kube-api-access-n596b\") pod \"payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z\" (UID: \"50c831dd-76a8-4e42-90b4-19847ae771e4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" Oct 29 23:24:31.642381 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.642337 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr"] Oct 29 23:24:31.671484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.651534 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" Oct 29 23:24:31.680713 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.680589 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr"] Oct 29 23:24:31.700755 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.700249 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c831dd-76a8-4e42-90b4-19847ae771e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z\" (UID: \"50c831dd-76a8-4e42-90b4-19847ae771e4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" Oct 29 23:24:31.700755 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.700328 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n596b\" (UniqueName: \"kubernetes.io/projected/50c831dd-76a8-4e42-90b4-19847ae771e4-kube-api-access-n596b\") pod \"payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z\" (UID: \"50c831dd-76a8-4e42-90b4-19847ae771e4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" Oct 29 23:24:31.701716 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.701678 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c831dd-76a8-4e42-90b4-19847ae771e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z\" (UID: \"50c831dd-76a8-4e42-90b4-19847ae771e4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" Oct 29 23:24:31.718348 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.718316 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n596b\" (UniqueName: \"kubernetes.io/projected/50c831dd-76a8-4e42-90b4-19847ae771e4-kube-api-access-n596b\") pod \"payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z\" (UID: \"50c831dd-76a8-4e42-90b4-19847ae771e4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" Oct 29 23:24:31.800880 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.800846 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f9be75-9161-4cd4-9b75-8428406eed15-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr\" (UID: \"10f9be75-9161-4cd4-9b75-8428406eed15\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" Oct 29 23:24:31.801087 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.801058 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8sk5\" (UniqueName: \"kubernetes.io/projected/10f9be75-9161-4cd4-9b75-8428406eed15-kube-api-access-n8sk5\") pod \"payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr\" (UID: \"10f9be75-9161-4cd4-9b75-8428406eed15\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" Oct 29 23:24:31.874090 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.874005 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" Oct 29 23:24:31.902764 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.902726 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8sk5\" (UniqueName: \"kubernetes.io/projected/10f9be75-9161-4cd4-9b75-8428406eed15-kube-api-access-n8sk5\") pod \"payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr\" (UID: \"10f9be75-9161-4cd4-9b75-8428406eed15\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" Oct 29 23:24:31.902929 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.902790 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f9be75-9161-4cd4-9b75-8428406eed15-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr\" (UID: \"10f9be75-9161-4cd4-9b75-8428406eed15\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" Oct 29 23:24:31.903625 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.903597 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f9be75-9161-4cd4-9b75-8428406eed15-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr\" (UID: \"10f9be75-9161-4cd4-9b75-8428406eed15\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" Oct 29 23:24:31.920709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:31.920682 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8sk5\" (UniqueName: \"kubernetes.io/projected/10f9be75-9161-4cd4-9b75-8428406eed15-kube-api-access-n8sk5\") pod \"payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr\" (UID: \"10f9be75-9161-4cd4-9b75-8428406eed15\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" Oct 29 23:24:32.006060 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.006027 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" Oct 29 23:24:32.172625 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.172548 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z"] Oct 29 23:24:32.175917 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:24:32.175878 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c831dd_76a8_4e42_90b4_19847ae771e4.slice/crio-23f2b98338c2fc18328049fb9ac7004132a0a1b44e1f9e1c2225f7b4f98c799c WatchSource:0}: Error finding container 23f2b98338c2fc18328049fb9ac7004132a0a1b44e1f9e1c2225f7b4f98c799c: Status 404 returned error can't find the container with id 23f2b98338c2fc18328049fb9ac7004132a0a1b44e1f9e1c2225f7b4f98c799c Oct 29 23:24:32.322771 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.322738 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" event={"ID":"50c831dd-76a8-4e42-90b4-19847ae771e4","Type":"ContainerStarted","Data":"23f2b98338c2fc18328049fb9ac7004132a0a1b44e1f9e1c2225f7b4f98c799c"} Oct 29 23:24:32.355193 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.355159 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr"] Oct 29 23:24:32.358493 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:24:32.358471 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f9be75_9161_4cd4_9b75_8428406eed15.slice/crio-7af60fd31a3086b0c72e4236f97d47315628b0d1ee1930463e89b00b725ed257 WatchSource:0}: Error finding container 7af60fd31a3086b0c72e4236f97d47315628b0d1ee1930463e89b00b725ed257: Status 404 returned error can't find the container with id 7af60fd31a3086b0c72e4236f97d47315628b0d1ee1930463e89b00b725ed257 Oct 29 23:24:32.605571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.605535 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg"] Oct 29 23:24:32.610536 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.610515 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" Oct 29 23:24:32.624083 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.624061 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-9280e4-dockercfg-fcr6m\"" Oct 29 23:24:32.629028 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.629003 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg"] Oct 29 23:24:32.673160 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.673115 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp"] Oct 29 23:24:32.676924 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.676787 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" Oct 29 23:24:32.695850 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.695812 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp"] Oct 29 23:24:32.702894 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.702868 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-9280e4\"" Oct 29 23:24:32.711273 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.711234 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80680fb1-ec38-4c40-8b92-ca49981bf54a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg\" (UID: \"80680fb1-ec38-4c40-8b92-ca49981bf54a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" Oct 29 23:24:32.711434 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.711295 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq5kg\" (UniqueName: \"kubernetes.io/projected/80680fb1-ec38-4c40-8b92-ca49981bf54a-kube-api-access-pq5kg\") pod \"payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg\" (UID: \"80680fb1-ec38-4c40-8b92-ca49981bf54a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" Oct 29 23:24:32.711434 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.711394 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77a9bc98-cb47-403d-b1ea-824dde7ae578-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp\" (UID: \"77a9bc98-cb47-403d-b1ea-824dde7ae578\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" Oct 29 23:24:32.711589 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.711493 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54qnf\" (UniqueName: \"kubernetes.io/projected/77a9bc98-cb47-403d-b1ea-824dde7ae578-kube-api-access-54qnf\") pod \"payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp\" (UID: \"77a9bc98-cb47-403d-b1ea-824dde7ae578\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" Oct 29 23:24:32.813144 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.813093 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77a9bc98-cb47-403d-b1ea-824dde7ae578-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp\" (UID: \"77a9bc98-cb47-403d-b1ea-824dde7ae578\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" Oct 29 23:24:32.814627 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.814590 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54qnf\" (UniqueName: \"kubernetes.io/projected/77a9bc98-cb47-403d-b1ea-824dde7ae578-kube-api-access-54qnf\") pod \"payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp\" (UID: \"77a9bc98-cb47-403d-b1ea-824dde7ae578\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" Oct 29 23:24:32.814804 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.814695 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80680fb1-ec38-4c40-8b92-ca49981bf54a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg\" (UID: \"80680fb1-ec38-4c40-8b92-ca49981bf54a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" Oct 29 23:24:32.814804 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.814739 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq5kg\" (UniqueName: \"kubernetes.io/projected/80680fb1-ec38-4c40-8b92-ca49981bf54a-kube-api-access-pq5kg\") pod \"payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg\" (UID: \"80680fb1-ec38-4c40-8b92-ca49981bf54a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" Oct 29 23:24:32.815820 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.815793 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77a9bc98-cb47-403d-b1ea-824dde7ae578-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp\" (UID: \"77a9bc98-cb47-403d-b1ea-824dde7ae578\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" Oct 29 23:24:32.816347 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.816324 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80680fb1-ec38-4c40-8b92-ca49981bf54a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg\" (UID: \"80680fb1-ec38-4c40-8b92-ca49981bf54a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" Oct 29 23:24:32.837991 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.837926 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq5kg\" (UniqueName: \"kubernetes.io/projected/80680fb1-ec38-4c40-8b92-ca49981bf54a-kube-api-access-pq5kg\") pod \"payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg\" (UID: \"80680fb1-ec38-4c40-8b92-ca49981bf54a\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" Oct 29 23:24:32.841114 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.841088 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54qnf\" (UniqueName: \"kubernetes.io/projected/77a9bc98-cb47-403d-b1ea-824dde7ae578-kube-api-access-54qnf\") pod \"payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp\" (UID: \"77a9bc98-cb47-403d-b1ea-824dde7ae578\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" Oct 29 23:24:32.923990 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.923895 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" Oct 29 23:24:32.996283 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:32.996250 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" Oct 29 23:24:33.255036 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.254945 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg"] Oct 29 23:24:33.264395 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:24:33.264355 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80680fb1_ec38_4c40_8b92_ca49981bf54a.slice/crio-f1a81f84d1845ceb50867eddf9a6b7811ce31ff84bb850ba4bbcefbbea6715c3 WatchSource:0}: Error finding container f1a81f84d1845ceb50867eddf9a6b7811ce31ff84bb850ba4bbcefbbea6715c3: Status 404 returned error can't find the container with id f1a81f84d1845ceb50867eddf9a6b7811ce31ff84bb850ba4bbcefbbea6715c3 Oct 29 23:24:33.330648 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.330589 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" event={"ID":"10f9be75-9161-4cd4-9b75-8428406eed15","Type":"ContainerStarted","Data":"7af60fd31a3086b0c72e4236f97d47315628b0d1ee1930463e89b00b725ed257"} Oct 29 23:24:33.332688 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.332648 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" event={"ID":"80680fb1-ec38-4c40-8b92-ca49981bf54a","Type":"ContainerStarted","Data":"f1a81f84d1845ceb50867eddf9a6b7811ce31ff84bb850ba4bbcefbbea6715c3"} Oct 29 23:24:33.345725 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.345681 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp"] Oct 29 23:24:33.358105 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:24:33.358066 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a9bc98_cb47_403d_b1ea_824dde7ae578.slice/crio-c5730d40d9fb90da35ea6ef5640621ad4fa2dde49ee72596d143d49baaaed5d6 WatchSource:0}: Error finding container c5730d40d9fb90da35ea6ef5640621ad4fa2dde49ee72596d143d49baaaed5d6: Status 404 returned error can't find the container with id c5730d40d9fb90da35ea6ef5640621ad4fa2dde49ee72596d143d49baaaed5d6 Oct 29 23:24:33.642529 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.642413 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p"] Oct 29 23:24:33.646637 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.646611 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" Oct 29 23:24:33.660304 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.660208 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-837245-dockercfg-p68lx\"" Oct 29 23:24:33.660304 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.660274 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-837245\"" Oct 29 23:24:33.671408 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.671372 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p"] Oct 29 23:24:33.726670 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.726625 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52x77\" (UniqueName: \"kubernetes.io/projected/cb462872-9968-4e91-96c9-675ffd093457-kube-api-access-52x77\") pod \"payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p\" (UID: \"cb462872-9968-4e91-96c9-675ffd093457\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" Oct 29 23:24:33.726845 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.726738 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb462872-9968-4e91-96c9-675ffd093457-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p\" (UID: \"cb462872-9968-4e91-96c9-675ffd093457\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" Oct 29 23:24:33.747038 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.746285 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg"] Oct 29 23:24:33.752536 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.751951 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" Oct 29 23:24:33.773534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.772191 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg"] Oct 29 23:24:33.827518 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.827448 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llqkr\" (UniqueName: \"kubernetes.io/projected/71038474-e817-4ff1-be48-145e6f10cb3c-kube-api-access-llqkr\") pod \"payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg\" (UID: \"71038474-e817-4ff1-be48-145e6f10cb3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" Oct 29 23:24:33.827691 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.827626 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52x77\" (UniqueName: \"kubernetes.io/projected/cb462872-9968-4e91-96c9-675ffd093457-kube-api-access-52x77\") pod \"payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p\" (UID: \"cb462872-9968-4e91-96c9-675ffd093457\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" Oct 29 23:24:33.827732 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.827708 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb462872-9968-4e91-96c9-675ffd093457-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p\" (UID: \"cb462872-9968-4e91-96c9-675ffd093457\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" Oct 29 23:24:33.828189 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.828162 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71038474-e817-4ff1-be48-145e6f10cb3c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg\" (UID: \"71038474-e817-4ff1-be48-145e6f10cb3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" Oct 29 23:24:33.828593 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.828573 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb462872-9968-4e91-96c9-675ffd093457-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p\" (UID: \"cb462872-9968-4e91-96c9-675ffd093457\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" Oct 29 23:24:33.845599 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.845562 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52x77\" (UniqueName: \"kubernetes.io/projected/cb462872-9968-4e91-96c9-675ffd093457-kube-api-access-52x77\") pod \"payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p\" (UID: \"cb462872-9968-4e91-96c9-675ffd093457\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" Oct 29 23:24:33.901893 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.901052 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:24:33.967522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.931080 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llqkr\" (UniqueName: \"kubernetes.io/projected/71038474-e817-4ff1-be48-145e6f10cb3c-kube-api-access-llqkr\") pod \"payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg\" (UID: \"71038474-e817-4ff1-be48-145e6f10cb3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" Oct 29 23:24:33.967522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.931247 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71038474-e817-4ff1-be48-145e6f10cb3c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg\" (UID: \"71038474-e817-4ff1-be48-145e6f10cb3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" Oct 29 23:24:33.967522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.932327 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71038474-e817-4ff1-be48-145e6f10cb3c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg\" (UID: \"71038474-e817-4ff1-be48-145e6f10cb3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" Oct 29 23:24:33.977505 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.968623 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" Oct 29 23:24:33.995007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:33.990851 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llqkr\" (UniqueName: \"kubernetes.io/projected/71038474-e817-4ff1-be48-145e6f10cb3c-kube-api-access-llqkr\") pod \"payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg\" (UID: \"71038474-e817-4ff1-be48-145e6f10cb3c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" Oct 29 23:24:34.082738 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:34.082476 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" Oct 29 23:24:34.349640 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:34.349594 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" event={"ID":"77a9bc98-cb47-403d-b1ea-824dde7ae578","Type":"ContainerStarted","Data":"c5730d40d9fb90da35ea6ef5640621ad4fa2dde49ee72596d143d49baaaed5d6"} Oct 29 23:24:34.427598 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:34.426928 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p"] Oct 29 23:24:34.435917 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:24:34.435878 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb462872_9968_4e91_96c9_675ffd093457.slice/crio-e2c4416dcc115b47d1d2845762eb3714d09066627a0d17939981b8b78bbdc8a9 WatchSource:0}: Error finding container e2c4416dcc115b47d1d2845762eb3714d09066627a0d17939981b8b78bbdc8a9: Status 404 returned error can't find the container with id e2c4416dcc115b47d1d2845762eb3714d09066627a0d17939981b8b78bbdc8a9 Oct 29 23:24:34.475841 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:34.475755 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg"] Oct 29 23:24:34.479001 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:24:34.478969 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71038474_e817_4ff1_be48_145e6f10cb3c.slice/crio-8a89a822c6905331746224a9bc7361a7a36c79b73316d9cff32cb94cf5caa5c1 WatchSource:0}: Error finding container 8a89a822c6905331746224a9bc7361a7a36c79b73316d9cff32cb94cf5caa5c1: Status 404 returned error can't find the container with id 8a89a822c6905331746224a9bc7361a7a36c79b73316d9cff32cb94cf5caa5c1 Oct 29 23:24:35.376974 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.376933 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" event={"ID":"cb462872-9968-4e91-96c9-675ffd093457","Type":"ContainerStarted","Data":"e2c4416dcc115b47d1d2845762eb3714d09066627a0d17939981b8b78bbdc8a9"} Oct 29 23:24:35.384764 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.384725 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" event={"ID":"71038474-e817-4ff1-be48-145e6f10cb3c","Type":"ContainerStarted","Data":"8a89a822c6905331746224a9bc7361a7a36c79b73316d9cff32cb94cf5caa5c1"} Oct 29 23:24:35.718506 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.717577 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2"] Oct 29 23:24:35.724078 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.724050 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" Oct 29 23:24:35.742923 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.742891 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2"] Oct 29 23:24:35.749481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.748956 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-2670a5-dockercfg-zzfbf\"" Oct 29 23:24:35.769245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.769203 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-2670a5\"" Oct 29 23:24:35.803415 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.803376 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2"] Oct 29 23:24:35.813068 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.812763 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" Oct 29 23:24:35.861541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.858616 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2"] Oct 29 23:24:35.882575 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.882538 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kct6j\" (UniqueName: \"kubernetes.io/projected/56b5ecf5-e109-433c-963d-bf97a5bd6f8a-kube-api-access-kct6j\") pod \"payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2\" (UID: \"56b5ecf5-e109-433c-963d-bf97a5bd6f8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" Oct 29 23:24:35.883004 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.882973 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b5ecf5-e109-433c-963d-bf97a5bd6f8a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2\" (UID: \"56b5ecf5-e109-433c-963d-bf97a5bd6f8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" Oct 29 23:24:35.984255 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.984168 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kct6j\" (UniqueName: \"kubernetes.io/projected/56b5ecf5-e109-433c-963d-bf97a5bd6f8a-kube-api-access-kct6j\") pod \"payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2\" (UID: \"56b5ecf5-e109-433c-963d-bf97a5bd6f8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" Oct 29 23:24:35.984527 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.984503 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czb8\" (UniqueName: \"kubernetes.io/projected/8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26-kube-api-access-4czb8\") pod \"payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2\" (UID: \"8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" Oct 29 23:24:35.984759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.984738 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2\" (UID: \"8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" Oct 29 23:24:35.984948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.984930 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b5ecf5-e109-433c-963d-bf97a5bd6f8a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2\" (UID: \"56b5ecf5-e109-433c-963d-bf97a5bd6f8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" Oct 29 23:24:35.986444 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:35.986423 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b5ecf5-e109-433c-963d-bf97a5bd6f8a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2\" (UID: \"56b5ecf5-e109-433c-963d-bf97a5bd6f8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" Oct 29 23:24:36.025356 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.025322 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kct6j\" (UniqueName: \"kubernetes.io/projected/56b5ecf5-e109-433c-963d-bf97a5bd6f8a-kube-api-access-kct6j\") pod \"payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2\" (UID: \"56b5ecf5-e109-433c-963d-bf97a5bd6f8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" Oct 29 23:24:36.067196 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.066927 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" Oct 29 23:24:36.088494 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.086338 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czb8\" (UniqueName: \"kubernetes.io/projected/8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26-kube-api-access-4czb8\") pod \"payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2\" (UID: \"8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" Oct 29 23:24:36.088494 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.086431 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2\" (UID: \"8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" Oct 29 23:24:36.088494 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.087337 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2\" (UID: \"8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" Oct 29 23:24:36.135632 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.135562 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czb8\" (UniqueName: \"kubernetes.io/projected/8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26-kube-api-access-4czb8\") pod \"payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2\" (UID: \"8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" Oct 29 23:24:36.178014 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.177976 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" Oct 29 23:24:36.434394 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.434356 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2"] Oct 29 23:24:36.440514 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:24:36.440467 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b5ecf5_e109_433c_963d_bf97a5bd6f8a.slice/crio-c4e9779b199c3b6aaadd150295da46c403e068354e6a33860c35a8e6b22662a1 WatchSource:0}: Error finding container c4e9779b199c3b6aaadd150295da46c403e068354e6a33860c35a8e6b22662a1: Status 404 returned error can't find the container with id c4e9779b199c3b6aaadd150295da46c403e068354e6a33860c35a8e6b22662a1 Oct 29 23:24:36.570949 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.570876 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2"] Oct 29 23:24:36.576635 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:24:36.576599 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb1736b_d3a6_4de0_bfb5_4a9ce2d90c26.slice/crio-0c54abe784dcc295efbe88f0c210e1a082f2fba155ba5d3319e159a0e85258d9 WatchSource:0}: Error finding container 0c54abe784dcc295efbe88f0c210e1a082f2fba155ba5d3319e159a0e85258d9: Status 404 returned error can't find the container with id 0c54abe784dcc295efbe88f0c210e1a082f2fba155ba5d3319e159a0e85258d9 Oct 29 23:24:36.889775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.889687 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7"] Oct 29 23:24:36.928188 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.895690 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" Oct 29 23:24:36.939004 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.938973 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-313d9c\"" Oct 29 23:24:36.940145 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.940118 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7"] Oct 29 23:24:36.965828 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.965753 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-313d9c-dockercfg-qc5fs\"" Oct 29 23:24:36.977531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.974514 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc"] Oct 29 23:24:36.984241 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.982799 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" Oct 29 23:24:36.999954 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:36.999927 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2t2r\" (UniqueName: \"kubernetes.io/projected/9c730892-1170-44b8-9fcf-e74a43035722-kube-api-access-g2t2r\") pod \"payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7\" (UID: \"9c730892-1170-44b8-9fcf-e74a43035722\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" Oct 29 23:24:37.000248 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.000226 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c730892-1170-44b8-9fcf-e74a43035722-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7\" (UID: \"9c730892-1170-44b8-9fcf-e74a43035722\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" Oct 29 23:24:37.006500 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.002491 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc"] Oct 29 23:24:37.133263 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.114999 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c730892-1170-44b8-9fcf-e74a43035722-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7\" (UID: \"9c730892-1170-44b8-9fcf-e74a43035722\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" Oct 29 23:24:37.133263 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.115127 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6173dedb-87b4-4028-9506-34ffadb93263-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc\" (UID: \"6173dedb-87b4-4028-9506-34ffadb93263\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" Oct 29 23:24:37.133263 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.115168 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dqnt\" (UniqueName: \"kubernetes.io/projected/6173dedb-87b4-4028-9506-34ffadb93263-kube-api-access-4dqnt\") pod \"payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc\" (UID: \"6173dedb-87b4-4028-9506-34ffadb93263\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" Oct 29 23:24:37.133263 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.115233 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2t2r\" (UniqueName: \"kubernetes.io/projected/9c730892-1170-44b8-9fcf-e74a43035722-kube-api-access-g2t2r\") pod \"payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7\" (UID: \"9c730892-1170-44b8-9fcf-e74a43035722\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" Oct 29 23:24:37.133263 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.132404 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c730892-1170-44b8-9fcf-e74a43035722-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7\" (UID: \"9c730892-1170-44b8-9fcf-e74a43035722\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" Oct 29 23:24:37.162890 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.162824 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2t2r\" (UniqueName: \"kubernetes.io/projected/9c730892-1170-44b8-9fcf-e74a43035722-kube-api-access-g2t2r\") pod \"payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7\" (UID: \"9c730892-1170-44b8-9fcf-e74a43035722\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" Oct 29 23:24:37.217632 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.216006 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dqnt\" (UniqueName: \"kubernetes.io/projected/6173dedb-87b4-4028-9506-34ffadb93263-kube-api-access-4dqnt\") pod \"payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc\" (UID: \"6173dedb-87b4-4028-9506-34ffadb93263\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" Oct 29 23:24:37.217632 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.216203 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6173dedb-87b4-4028-9506-34ffadb93263-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc\" (UID: \"6173dedb-87b4-4028-9506-34ffadb93263\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" Oct 29 23:24:37.217632 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.217268 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6173dedb-87b4-4028-9506-34ffadb93263-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc\" (UID: \"6173dedb-87b4-4028-9506-34ffadb93263\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" Oct 29 23:24:37.240327 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.240228 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dqnt\" (UniqueName: \"kubernetes.io/projected/6173dedb-87b4-4028-9506-34ffadb93263-kube-api-access-4dqnt\") pod \"payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc\" (UID: \"6173dedb-87b4-4028-9506-34ffadb93263\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" Oct 29 23:24:37.258410 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.258152 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" Oct 29 23:24:37.333965 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.333662 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" Oct 29 23:24:37.443424 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.443392 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" event={"ID":"8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26","Type":"ContainerStarted","Data":"90da7b065cc3fdb3456ce9da4ca0713123a51c97766fc4b481522497c07c9d05"} Oct 29 23:24:37.444153 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.443473 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" event={"ID":"8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26","Type":"ContainerStarted","Data":"0c54abe784dcc295efbe88f0c210e1a082f2fba155ba5d3319e159a0e85258d9"} Oct 29 23:24:37.444390 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.444359 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" Oct 29 23:24:37.447545 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.447517 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" event={"ID":"50c831dd-76a8-4e42-90b4-19847ae771e4","Type":"ContainerStarted","Data":"0ebd4baad9857e2bd2e0580783ff3ee5b2953d22c1239a124f1ff8b08ac10097"} Oct 29 23:24:37.448144 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.448121 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" Oct 29 23:24:37.452508 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.452166 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" event={"ID":"10f9be75-9161-4cd4-9b75-8428406eed15","Type":"ContainerStarted","Data":"3cc034b607740335b7b1c3dbdbdb80110f3640cb10d1bc98d4e6bd3a6a41bda3"} Oct 29 23:24:37.453129 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.452959 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" Oct 29 23:24:37.456480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.456341 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" event={"ID":"56b5ecf5-e109-433c-963d-bf97a5bd6f8a","Type":"ContainerStarted","Data":"cf43714541cc6be5c9272430fee0b8b1f373d7038cd504c588f67aad97f3d52a"} Oct 29 23:24:37.456480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.456382 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" event={"ID":"56b5ecf5-e109-433c-963d-bf97a5bd6f8a","Type":"ContainerStarted","Data":"c4e9779b199c3b6aaadd150295da46c403e068354e6a33860c35a8e6b22662a1"} Oct 29 23:24:37.530518 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.527669 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" podStartSLOduration=2.2004548760000002 podStartE2EDuration="2.527649741s" totalImagesPullingTime="327.194865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:37.525929155 +0000 UTC m=+646.888933879" watchObservedRunningTime="2026-10-29 23:24:37.527649741 +0000 UTC m=+646.890654466" Oct 29 23:24:37.532012 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.531951 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" podStartSLOduration=2.24428597 podStartE2EDuration="2.53193556s" totalImagesPullingTime="287.64959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:37.480650755 +0000 UTC m=+646.843655475" watchObservedRunningTime="2026-10-29 23:24:37.53193556 +0000 UTC m=+646.894940318" Oct 29 23:24:37.559339 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.559286 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" podStartSLOduration=1.99989368 podStartE2EDuration="6.559272271s" totalImagesPullingTime="4.559378591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:37.557937879 +0000 UTC m=+646.920942601" watchObservedRunningTime="2026-10-29 23:24:37.559272271 +0000 UTC m=+646.922276994" Oct 29 23:24:37.594830 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.594731 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" podStartSLOduration=1.854849839 podStartE2EDuration="6.594718096s" totalImagesPullingTime="4.739868257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:37.591494457 +0000 UTC m=+646.954499181" watchObservedRunningTime="2026-10-29 23:24:37.594718096 +0000 UTC m=+646.957722819" Oct 29 23:24:37.596712 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.596675 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7"] Oct 29 23:24:37.691354 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:37.691309 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc"] Oct 29 23:24:37.697258 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:24:37.697230 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6173dedb_87b4_4028_9506_34ffadb93263.slice/crio-244373564e5e8615dada25569891bb2fd3de777ccbbaccc5873d556c1f352054 WatchSource:0}: Error finding container 244373564e5e8615dada25569891bb2fd3de777ccbbaccc5873d556c1f352054: Status 404 returned error can't find the container with id 244373564e5e8615dada25569891bb2fd3de777ccbbaccc5873d556c1f352054 Oct 29 23:24:38.465080 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:38.464196 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" event={"ID":"9c730892-1170-44b8-9fcf-e74a43035722","Type":"ContainerStarted","Data":"d3b14bd874fd59a3a847a785864271dc7273082d9b3253319e41e78f2cbb0f0d"} Oct 29 23:24:38.465080 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:38.464264 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" event={"ID":"9c730892-1170-44b8-9fcf-e74a43035722","Type":"ContainerStarted","Data":"9f4f1fa5b9587d8aec3b3f9aac6cdfb8ade2c3a882c7b16c0f8c4f8380bc2979"} Oct 29 23:24:38.465080 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:38.465027 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" Oct 29 23:24:38.467212 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:38.467178 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" event={"ID":"cb462872-9968-4e91-96c9-675ffd093457","Type":"ContainerStarted","Data":"750047c8bae361f192f8e2f5e12aec9a9bb1fff05002b40f98186bf16b0fb282"} Oct 29 23:24:38.467627 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:38.467611 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" Oct 29 23:24:38.468999 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:38.468976 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" event={"ID":"6173dedb-87b4-4028-9506-34ffadb93263","Type":"ContainerStarted","Data":"50615357a1b5beac4da23d76a90f5e586e298be3251d9682c006396ef94685ec"} Oct 29 23:24:38.469101 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:38.469014 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" event={"ID":"6173dedb-87b4-4028-9506-34ffadb93263","Type":"ContainerStarted","Data":"244373564e5e8615dada25569891bb2fd3de777ccbbaccc5873d556c1f352054"} Oct 29 23:24:38.469101 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:38.469076 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" Oct 29 23:24:38.470308 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:38.470286 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" event={"ID":"71038474-e817-4ff1-be48-145e6f10cb3c","Type":"ContainerStarted","Data":"b23794173ebc514c6f2e63ad497e1aa15c01986f8039a44e2e63961c5398b5cd"} Oct 29 23:24:38.470667 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:38.470650 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" Oct 29 23:24:38.470837 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:38.470821 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" Oct 29 23:24:38.498128 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:38.498072 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" podStartSLOduration=2.498045852 podStartE2EDuration="2.498045852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:38.496355555 +0000 UTC m=+647.859360270" watchObservedRunningTime="2026-10-29 23:24:38.498045852 +0000 UTC m=+647.861050581" Oct 29 23:24:38.535297 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:38.535206 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" podStartSLOduration=2.079193292 podStartE2EDuration="5.535190433s" totalImagesPullingTime="3.455997141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:38.534111379 +0000 UTC m=+647.897116096" watchObservedRunningTime="2026-10-29 23:24:38.535190433 +0000 UTC m=+647.898195157" Oct 29 23:24:38.573537 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:38.573444 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" podStartSLOduration=1.985758504 podStartE2EDuration="5.573426293s" totalImagesPullingTime="3.587667789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:38.569138856 +0000 UTC m=+647.932143611" watchObservedRunningTime="2026-10-29 23:24:38.573426293 +0000 UTC m=+647.936431017" Oct 29 23:24:38.629481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:38.627567 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" podStartSLOduration=2.627548592 podStartE2EDuration="2.627548592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:38.625138654 +0000 UTC m=+647.988143377" watchObservedRunningTime="2026-10-29 23:24:38.627548592 +0000 UTC m=+647.990553315" Oct 29 23:24:39.023539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.020844 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:24:39.483575 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.483531 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" event={"ID":"77a9bc98-cb47-403d-b1ea-824dde7ae578","Type":"ContainerStarted","Data":"e30269bd21b2a956143814f669d70a262597d660daf9e25a09925e1985df8888"} Oct 29 23:24:39.484065 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.483609 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" Oct 29 23:24:39.485989 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.485962 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" event={"ID":"80680fb1-ec38-4c40-8b92-ca49981bf54a","Type":"ContainerStarted","Data":"860f1424f34b78fe28cfcccb0a16b18c226f77cabfa17d718e773fc48fc4780d"} Oct 29 23:24:39.486512 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.486428 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" Oct 29 23:24:39.558903 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.558842 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" podStartSLOduration=2.176706876 podStartE2EDuration="7.558827678s" totalImagesPullingTime="5.382120802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:39.556629685 +0000 UTC m=+648.919634407" watchObservedRunningTime="2026-10-29 23:24:39.558827678 +0000 UTC m=+648.921832400" Oct 29 23:24:39.559164 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.559128 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" podStartSLOduration=2.386352924 podStartE2EDuration="7.559116859s" totalImagesPullingTime="5.172763935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:39.521059336 +0000 UTC m=+648.884064061" watchObservedRunningTime="2026-10-29 23:24:39.559116859 +0000 UTC m=+648.922121579" Oct 29 23:24:39.875971 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.875881 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st"] Oct 29 23:24:39.880577 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.880549 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" Oct 29 23:24:39.896312 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.896290 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-0b2ae9\"" Oct 29 23:24:39.936509 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.936383 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st"] Oct 29 23:24:39.951031 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.951002 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn"] Oct 29 23:24:39.954970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.954948 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" Oct 29 23:24:39.966990 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.966968 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-0b2ae9-dockercfg-7fxqj\"" Oct 29 23:24:39.975879 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.975856 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn"] Oct 29 23:24:39.980747 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.980719 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee03dd91-8ebd-4268-9fbc-9017f73223b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st\" (UID: \"ee03dd91-8ebd-4268-9fbc-9017f73223b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" Oct 29 23:24:39.980946 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:39.980924 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kd27\" (UniqueName: \"kubernetes.io/projected/ee03dd91-8ebd-4268-9fbc-9017f73223b0-kube-api-access-5kd27\") pod \"payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st\" (UID: \"ee03dd91-8ebd-4268-9fbc-9017f73223b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" Oct 29 23:24:40.086343 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:40.086300 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74xj6\" (UniqueName: \"kubernetes.io/projected/55ba2c05-8056-4ccc-beeb-118dff6c1875-kube-api-access-74xj6\") pod \"payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn\" (UID: \"55ba2c05-8056-4ccc-beeb-118dff6c1875\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" Oct 29 23:24:40.086744 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:40.086719 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kd27\" (UniqueName: \"kubernetes.io/projected/ee03dd91-8ebd-4268-9fbc-9017f73223b0-kube-api-access-5kd27\") pod \"payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st\" (UID: \"ee03dd91-8ebd-4268-9fbc-9017f73223b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" Oct 29 23:24:40.087302 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:40.087281 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee03dd91-8ebd-4268-9fbc-9017f73223b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st\" (UID: \"ee03dd91-8ebd-4268-9fbc-9017f73223b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" Oct 29 23:24:40.088434 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:40.088413 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55ba2c05-8056-4ccc-beeb-118dff6c1875-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn\" (UID: \"55ba2c05-8056-4ccc-beeb-118dff6c1875\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" Oct 29 23:24:40.088668 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:40.088274 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee03dd91-8ebd-4268-9fbc-9017f73223b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st\" (UID: \"ee03dd91-8ebd-4268-9fbc-9017f73223b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" Oct 29 23:24:40.130197 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:40.130125 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kd27\" (UniqueName: \"kubernetes.io/projected/ee03dd91-8ebd-4268-9fbc-9017f73223b0-kube-api-access-5kd27\") pod \"payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st\" (UID: \"ee03dd91-8ebd-4268-9fbc-9017f73223b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" Oct 29 23:24:40.189720 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:40.189679 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74xj6\" (UniqueName: \"kubernetes.io/projected/55ba2c05-8056-4ccc-beeb-118dff6c1875-kube-api-access-74xj6\") pod \"payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn\" (UID: \"55ba2c05-8056-4ccc-beeb-118dff6c1875\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" Oct 29 23:24:40.190448 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:40.190424 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55ba2c05-8056-4ccc-beeb-118dff6c1875-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn\" (UID: \"55ba2c05-8056-4ccc-beeb-118dff6c1875\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" Oct 29 23:24:40.191200 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:40.191178 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55ba2c05-8056-4ccc-beeb-118dff6c1875-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn\" (UID: \"55ba2c05-8056-4ccc-beeb-118dff6c1875\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" Oct 29 23:24:40.194668 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:40.194646 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" Oct 29 23:24:40.226162 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:40.226125 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74xj6\" (UniqueName: \"kubernetes.io/projected/55ba2c05-8056-4ccc-beeb-118dff6c1875-kube-api-access-74xj6\") pod \"payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn\" (UID: \"55ba2c05-8056-4ccc-beeb-118dff6c1875\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" Oct 29 23:24:40.276708 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:40.272655 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" Oct 29 23:24:40.866710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:40.865987 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st"] Oct 29 23:24:41.027771 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:41.025835 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn"] Oct 29 23:24:41.557539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:41.556957 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" event={"ID":"55ba2c05-8056-4ccc-beeb-118dff6c1875","Type":"ContainerStarted","Data":"cb339fcf1cdfae9cb8c567d1841f8cb599fefedb8ee21fb04a65b53a850408eb"} Oct 29 23:24:41.557539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:41.557021 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" event={"ID":"55ba2c05-8056-4ccc-beeb-118dff6c1875","Type":"ContainerStarted","Data":"80007441a067ec52134557fdb1251561669a2a1851c1e2ba511e02afc51157e4"} Oct 29 23:24:41.557539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:41.557080 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" Oct 29 23:24:41.561480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:41.559009 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" event={"ID":"ee03dd91-8ebd-4268-9fbc-9017f73223b0","Type":"ContainerStarted","Data":"2a0e1f64eddc36d0540327e09c054b18c06a7007ab4dfd76302b7e9035d1d90e"} Oct 29 23:24:41.561480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:41.559074 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" event={"ID":"ee03dd91-8ebd-4268-9fbc-9017f73223b0","Type":"ContainerStarted","Data":"90300eb4368410880eba441b8d59fa6bb8a31da14c0484339a374adca1626072"} Oct 29 23:24:41.567344 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:41.567254 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" Oct 29 23:24:41.598233 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:41.598186 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" podStartSLOduration=2.598173161 podStartE2EDuration="2.598173161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:41.597851595 +0000 UTC m=+650.960856317" watchObservedRunningTime="2026-10-29 23:24:41.598173161 +0000 UTC m=+650.961177884" Oct 29 23:24:41.638205 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:41.638138 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" podStartSLOduration=2.638117898 podStartE2EDuration="2.638117898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:41.637719958 +0000 UTC m=+651.000724685" watchObservedRunningTime="2026-10-29 23:24:41.638117898 +0000 UTC m=+651.001122625" Oct 29 23:24:43.344484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:43.340585 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:24:43.344484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:43.340654 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" podUID="aa280316-fb6c-40db-8658-65b84bb4382b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:24:43.715482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:43.709184 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:24:45.276231 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:24:45.276174 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:24:46.592568 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:46.592530 6507 generic.go:356] "Generic (PLEG): container finished" podID="aa280316-fb6c-40db-8658-65b84bb4382b" containerID="369f12f078ada2771f8e0a12bb8823e4ef31052e6dede14826fd54975da29523" exitCode=0 Oct 29 23:24:46.593093 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:46.592580 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" event={"ID":"aa280316-fb6c-40db-8658-65b84bb4382b","Type":"ContainerDied","Data":"369f12f078ada2771f8e0a12bb8823e4ef31052e6dede14826fd54975da29523"} Oct 29 23:24:46.593093 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:46.593026 6507 scope.go:122] "RemoveContainer" containerID="369f12f078ada2771f8e0a12bb8823e4ef31052e6dede14826fd54975da29523" Oct 29 23:24:46.594687 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:46.594653 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:46.594853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:46.594749 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:46.594853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:46.594803 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:47.607484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:47.601446 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" event={"ID":"aa280316-fb6c-40db-8658-65b84bb4382b","Type":"ContainerStarted","Data":"d6a25edbc79d32ff0fdc5947f6f45dfd61e520e49b93b29de043754e00bab8bd"} Oct 29 23:24:47.658288 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:47.658254 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:47.658573 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:47.658554 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 29 23:24:47.658679 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:47.658597 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" podUID="aa280316-fb6c-40db-8658-65b84bb4382b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 29 23:24:48.658690 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:48.658656 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 29 23:24:48.659113 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:48.658718 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" podUID="aa280316-fb6c-40db-8658-65b84bb4382b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 29 23:24:49.489555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:49.489523 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" Oct 29 23:24:49.489739 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:49.489612 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" Oct 29 23:24:49.489739 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:49.489660 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" Oct 29 23:24:49.490043 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:49.490022 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" Oct 29 23:24:49.490683 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:49.490664 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" Oct 29 23:24:49.491304 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:49.491286 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" Oct 29 23:24:49.657721 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:49.657681 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 29 23:24:49.660011 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:49.657742 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" podUID="aa280316-fb6c-40db-8658-65b84bb4382b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 29 23:24:49.864357 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:49.863932 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:24:50.517250 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.517160 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" Oct 29 23:24:50.526483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.522811 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" Oct 29 23:24:50.526844 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.526819 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" Oct 29 23:24:50.543482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.540647 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" Oct 29 23:24:50.657739 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.657701 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 29 23:24:50.657944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.657777 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" podUID="aa280316-fb6c-40db-8658-65b84bb4382b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 29 23:24:50.658188 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.658161 6507 generic.go:356] "Generic (PLEG): container finished" podID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" containerID="a48503f9e10bea7b5196858f404217cf7b5d7b2bb0c153df2c0030ad1748b171" exitCode=0 Oct 29 23:24:50.658253 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.658235 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" event={"ID":"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2","Type":"ContainerDied","Data":"a48503f9e10bea7b5196858f404217cf7b5d7b2bb0c153df2c0030ad1748b171"} Oct 29 23:24:50.658680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.658659 6507 scope.go:122] "RemoveContainer" containerID="a48503f9e10bea7b5196858f404217cf7b5d7b2bb0c153df2c0030ad1748b171" Oct 29 23:24:50.660079 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.660058 6507 generic.go:356] "Generic (PLEG): container finished" podID="6bf4f192-0296-4a17-abf7-b6a55b12f60c" containerID="21ac6a31e98cbcf718f5c10e510b40b8fda554c41f516d4b6842a0a56a583170" exitCode=0 Oct 29 23:24:50.660446 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.660137 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" event={"ID":"6bf4f192-0296-4a17-abf7-b6a55b12f60c","Type":"ContainerDied","Data":"21ac6a31e98cbcf718f5c10e510b40b8fda554c41f516d4b6842a0a56a583170"} Oct 29 23:24:50.660446 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.660162 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:50.660446 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.660251 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:50.660446 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.660300 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:50.660446 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.660444 6507 scope.go:122] "RemoveContainer" containerID="21ac6a31e98cbcf718f5c10e510b40b8fda554c41f516d4b6842a0a56a583170" Oct 29 23:24:50.661812 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.661781 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:50.661911 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.661863 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:50.661966 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:50.661910 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:24:51.657979 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:51.657936 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 29 23:24:51.658761 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:51.658244 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" podUID="aa280316-fb6c-40db-8658-65b84bb4382b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 29 23:24:51.693345 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:51.693307 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" event={"ID":"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2","Type":"ContainerStarted","Data":"7e0c75458f20a828c3ebf7997e6f3845f0256d76897ad557ee7a9b95aa5ca6d0"} Oct 29 23:24:51.705250 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:51.705205 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" event={"ID":"6bf4f192-0296-4a17-abf7-b6a55b12f60c","Type":"ContainerStarted","Data":"c36f2552dece2633e15fc2be06bb4b920a1aad6d558614c333798c005faaff69"} Oct 29 23:24:51.826482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:51.818520 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:51.829279 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:51.829245 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 29 23:24:51.829553 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:51.829527 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" podUID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 29 23:24:51.841896 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:51.841864 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:51.845993 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:51.845957 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 29 23:24:51.846242 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:51.846214 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" podUID="6bf4f192-0296-4a17-abf7-b6a55b12f60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 29 23:24:52.613483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:52.594686 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" Oct 29 23:24:52.780495 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:52.753982 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 29 23:24:52.781498 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:52.781428 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" podUID="aa280316-fb6c-40db-8658-65b84bb4382b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 29 23:24:52.854696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:52.854659 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 29 23:24:52.854988 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:52.854957 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" podUID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 29 23:24:52.866684 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:52.866585 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 29 23:24:52.866977 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:52.866941 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" podUID="6bf4f192-0296-4a17-abf7-b6a55b12f60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 29 23:24:53.591013 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:53.590781 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" Oct 29 23:24:53.839482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:53.838944 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 29 23:24:53.839482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:53.839002 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" podUID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 29 23:24:53.846586 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:53.840420 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 29 23:24:53.846586 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:53.840489 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" podUID="6bf4f192-0296-4a17-abf7-b6a55b12f60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 29 23:24:54.229507 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:54.228805 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:54.230088 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:54.230060 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:54.281801 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:54.281762 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:24:54.833483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:54.832605 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 29 23:24:54.833483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:54.832689 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" podUID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 29 23:24:54.846053 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:54.846009 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 29 23:24:54.846568 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:54.846076 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" podUID="6bf4f192-0296-4a17-abf7-b6a55b12f60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 29 23:24:55.811484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:55.810888 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 29 23:24:55.811484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:55.810959 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" podUID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 29 23:24:55.855486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:55.852863 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 29 23:24:55.855486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:55.852925 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" podUID="6bf4f192-0296-4a17-abf7-b6a55b12f60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 29 23:24:56.292900 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:56.292850 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:56.822484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:56.821057 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 29 23:24:56.822484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:56.821165 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" podUID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 29 23:24:56.843484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:56.842373 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 29 23:24:56.843484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:56.842434 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" podUID="6bf4f192-0296-4a17-abf7-b6a55b12f60c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 29 23:24:57.298307 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:57.297394 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:57.816192 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:57.816146 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:57.903053 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:57.903014 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:57.904844 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:57.904806 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:24:58.014835 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:58.014803 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:24:59.189808 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:24:59.189766 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:25:02.036661 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:02.036631 6507 generic.go:356] "Generic (PLEG): container finished" podID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerID="606ae1be70bbb339138254abb97a9ce15e8334b24e3344ff19e42a6c40ff9567" exitCode=0 Oct 29 23:25:02.037114 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:02.036707 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" event={"ID":"dac751e7-bb4e-4bf7-81f2-26fee5a478f3","Type":"ContainerDied","Data":"606ae1be70bbb339138254abb97a9ce15e8334b24e3344ff19e42a6c40ff9567"} Oct 29 23:25:02.037114 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:02.037079 6507 scope.go:122] "RemoveContainer" containerID="606ae1be70bbb339138254abb97a9ce15e8334b24e3344ff19e42a6c40ff9567" Oct 29 23:25:02.038344 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:02.038318 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:25:02.038399 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:02.038382 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:25:02.038442 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:02.038415 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:25:03.044159 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:03.044125 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" event={"ID":"dac751e7-bb4e-4bf7-81f2-26fee5a478f3","Type":"ContainerStarted","Data":"cc5383e2bac9805fb9dd29aca5128fa5132932aad0f3e6ec04ee74428f8a6acb"} Oct 29 23:25:03.126745 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:03.126683 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:25:03.126964 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:03.126943 6507 patch_prober.go:29] interesting pod/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 29 23:25:03.127041 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:03.126998 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" podUID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 29 23:25:04.126939 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:04.126898 6507 patch_prober.go:29] interesting pod/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 29 23:25:04.127357 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:04.126967 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" podUID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 29 23:25:05.125022 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:05.124983 6507 patch_prober.go:29] interesting pod/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 29 23:25:05.126245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:05.126211 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" podUID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 29 23:25:06.126944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:06.126898 6507 patch_prober.go:29] interesting pod/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 29 23:25:06.127364 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:06.126988 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" podUID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 29 23:25:07.126944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:07.126905 6507 patch_prober.go:29] interesting pod/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 29 23:25:07.127408 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:07.126965 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" podUID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 29 23:25:08.124806 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:08.124767 6507 patch_prober.go:29] interesting pod/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 29 23:25:08.124974 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:08.124826 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" podUID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 29 23:25:09.128981 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:09.128946 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:25:09.129421 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:09.129346 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:25:09.130033 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:09.130013 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:25:11.015284 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:11.015242 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:25:12.465134 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.465096 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc"] Oct 29 23:25:12.471841 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.471810 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.486931 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.486898 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-313d9c\"/\"kube-root-ca.crt\"" Oct 29 23:25:12.496018 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.495985 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-313d9c\"/\"default-dockercfg-5mnf7\"" Oct 29 23:25:12.496575 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.496514 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-313d9c\"/\"simulator-w1-313d9c-kserve-self-signed-certs\"" Oct 29 23:25:12.496721 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.496607 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-313d9c\"/\"openshift-service-ca.crt\"" Oct 29 23:25:12.499663 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.499639 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc"] Oct 29 23:25:12.581135 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.581096 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-tmp-dir\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.581135 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.581141 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-dshm\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.581412 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.581173 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f27f775-181e-4522-b34e-10374fc64c45-tls-certs\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.581412 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.581308 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76xcx\" (UniqueName: \"kubernetes.io/projected/1f27f775-181e-4522-b34e-10374fc64c45-kube-api-access-76xcx\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.581556 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.581433 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-home\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.581601 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.581545 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-model-cache\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.684697 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.684658 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f27f775-181e-4522-b34e-10374fc64c45-tls-certs\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.684977 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.684951 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76xcx\" (UniqueName: \"kubernetes.io/projected/1f27f775-181e-4522-b34e-10374fc64c45-kube-api-access-76xcx\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.685176 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.685158 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-home\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.685315 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.685299 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-model-cache\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.685586 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.685564 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-tmp-dir\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.685745 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.685728 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-dshm\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.686927 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.686872 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-tmp-dir\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.695652 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.687203 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-home\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.704065 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.704035 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-model-cache\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.711482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.705095 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-dshm\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.711482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.711393 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f27f775-181e-4522-b34e-10374fc64c45-tls-certs\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.732528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.732188 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76xcx\" (UniqueName: \"kubernetes.io/projected/1f27f775-181e-4522-b34e-10374fc64c45-kube-api-access-76xcx\") pod \"simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:12.793209 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:12.793169 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:13.091660 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:13.091566 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc"] Oct 29 23:25:13.095913 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:25:13.095884 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f27f775_181e_4522_b34e_10374fc64c45.slice/crio-b79160fadbc65ba3c4f403c62f37338785a779cc0c02635a32be6114b5f022b0 WatchSource:0}: Error finding container b79160fadbc65ba3c4f403c62f37338785a779cc0c02635a32be6114b5f022b0: Status 404 returned error can't find the container with id b79160fadbc65ba3c4f403c62f37338785a779cc0c02635a32be6114b5f022b0 Oct 29 23:25:13.115211 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:13.115180 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" event={"ID":"1f27f775-181e-4522-b34e-10374fc64c45","Type":"ContainerStarted","Data":"b79160fadbc65ba3c4f403c62f37338785a779cc0c02635a32be6114b5f022b0"} Oct 29 23:25:14.122376 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:14.122336 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" event={"ID":"1f27f775-181e-4522-b34e-10374fc64c45","Type":"ContainerStarted","Data":"7506fa9074430cad8569770fb2b15c1500e5f41e87d1f9cf592f52bd91566bc6"} Oct 29 23:25:14.157065 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:14.156995 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podStartSLOduration=2.156974709 podStartE2EDuration="2.156974709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:14.155919695 +0000 UTC m=+683.518924397" watchObservedRunningTime="2026-10-29 23:25:14.156974709 +0000 UTC m=+683.519979433" Oct 29 23:25:14.793911 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:14.793872 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:15.223913 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:25:15.223878 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:25:22.794035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:22.793985 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:22.806540 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:22.806513 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:23.181284 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:23.181204 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:25:23.514163 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:23.514128 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:25:23.872975 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:23.872890 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx"] Oct 29 23:25:23.877045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:23.877014 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:23.897919 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:23.897897 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-032fa8-openshift-default-dockercfg-kpn7m\"" Oct 29 23:25:23.904392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:23.904366 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx"] Oct 29 23:25:24.027046 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.027009 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-credential-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.027212 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.027049 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-workload-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.027212 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.027163 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istiod-ca-cert\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.027289 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.027213 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-data\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.027323 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.027303 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gxlx\" (UniqueName: \"kubernetes.io/projected/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-kube-api-access-7gxlx\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.027361 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.027345 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-envoy\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.027394 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.027365 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-podinfo\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.027426 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.027415 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-workload-certs\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.027494 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.027440 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-token\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.129012 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.128915 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-workload-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.129012 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.129009 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istiod-ca-cert\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.129209 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.129052 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-data\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.129209 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.129096 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gxlx\" (UniqueName: \"kubernetes.io/projected/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-kube-api-access-7gxlx\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.129209 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.129115 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-envoy\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.129209 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.129131 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-podinfo\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.129402 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.129318 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-workload-certs\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.129402 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.129364 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-token\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.129608 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.129439 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-credential-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.129608 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.129506 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-data\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.129608 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.129364 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-workload-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.129853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.129825 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istiod-ca-cert\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.129972 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.129854 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-workload-certs\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.129972 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.129901 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-credential-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.131610 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.131587 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-envoy\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.131736 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.131688 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-podinfo\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.145597 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.145574 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-token\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.145741 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.145648 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gxlx\" (UniqueName: \"kubernetes.io/projected/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-kube-api-access-7gxlx\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.214482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.213679 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:24.561383 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.561352 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx"] Oct 29 23:25:24.564069 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:25:24.564033 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a70a55_4e5b_455a_b39f_e6e1b2489d03.slice/crio-102a04eb82cd0acce69d737adc142b335c731cae8e763affbe40b60cb6087900 WatchSource:0}: Error finding container 102a04eb82cd0acce69d737adc142b335c731cae8e763affbe40b60cb6087900: Status 404 returned error can't find the container with id 102a04eb82cd0acce69d737adc142b335c731cae8e763affbe40b60cb6087900 Oct 29 23:25:24.568682 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.568636 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:25:24.568791 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.568727 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:25:24.568791 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:24.568775 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:25:25.185895 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:25.185853 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" event={"ID":"d5a70a55-4e5b-455a-b39f-e6e1b2489d03","Type":"ContainerStarted","Data":"fa6e180ed393c7afd77b075bfdc4ace3e09a6ed4f5cd3b7c40c1ebb2ff42a52e"} Oct 29 23:25:25.186288 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:25.185920 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" event={"ID":"d5a70a55-4e5b-455a-b39f-e6e1b2489d03","Type":"ContainerStarted","Data":"102a04eb82cd0acce69d737adc142b335c731cae8e763affbe40b60cb6087900"} Oct 29 23:25:25.217480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:25.215845 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:25.220429 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:25.220396 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 29 23:25:25.220709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:25.220680 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" podUID="d5a70a55-4e5b-455a-b39f-e6e1b2489d03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 29 23:25:25.237193 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:25.237136 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" podStartSLOduration=2.23711735 podStartE2EDuration="2.23711735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:25.23600567 +0000 UTC m=+694.599010393" watchObservedRunningTime="2026-10-29 23:25:25.23711735 +0000 UTC m=+694.600122077" Oct 29 23:25:26.221483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:26.219878 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 29 23:25:26.221483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:26.219948 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" podUID="d5a70a55-4e5b-455a-b39f-e6e1b2489d03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 29 23:25:27.187061 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.187026 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf"] Oct 29 23:25:27.192157 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.191647 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.223482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.214727 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 29 23:25:27.223482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.214803 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" podUID="d5a70a55-4e5b-455a-b39f-e6e1b2489d03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 29 23:25:27.234014 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.233619 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-032fa8\"/\"openshift-service-ca.crt\"" Oct 29 23:25:27.234014 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.233759 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-032fa8\"/\"default-dockercfg-8rt48\"" Oct 29 23:25:27.234014 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.233848 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-032fa8\"/\"premium-w4-032fa8-kserve-self-signed-certs\"" Oct 29 23:25:27.234014 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.233924 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-032fa8\"/\"kube-root-ca.crt\"" Oct 29 23:25:27.236195 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.236168 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf"] Oct 29 23:25:27.368273 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.368222 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6gm8\" (UniqueName: \"kubernetes.io/projected/045c37c8-57a0-4ed1-87a6-89f73eb079d2-kube-api-access-x6gm8\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.368676 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.368629 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-home\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.368895 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.368875 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-model-cache\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.369037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.369021 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-tmp-dir\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.369208 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.369193 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-dshm\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.380828 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.375294 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/045c37c8-57a0-4ed1-87a6-89f73eb079d2-tls-certs\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.481954 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.481859 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/045c37c8-57a0-4ed1-87a6-89f73eb079d2-tls-certs\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.481954 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.481937 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6gm8\" (UniqueName: \"kubernetes.io/projected/045c37c8-57a0-4ed1-87a6-89f73eb079d2-kube-api-access-x6gm8\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.482185 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.481969 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-home\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.482185 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.481991 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-model-cache\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.482185 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.482120 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-tmp-dir\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.482185 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.482152 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-dshm\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.482439 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.482414 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-model-cache\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.482515 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.482441 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-tmp-dir\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.482687 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.482672 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-home\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.484300 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.484277 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/045c37c8-57a0-4ed1-87a6-89f73eb079d2-tls-certs\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.484500 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.484479 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-dshm\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.498990 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.498952 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6gm8\" (UniqueName: \"kubernetes.io/projected/045c37c8-57a0-4ed1-87a6-89f73eb079d2-kube-api-access-x6gm8\") pod \"premium-w4-032fa8-kserve-c674d5b7d-sv4kf\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.542624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.542587 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:27.852332 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:27.852204 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf"] Oct 29 23:25:27.856054 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:25:27.856027 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod045c37c8_57a0_4ed1_87a6_89f73eb079d2.slice/crio-0fde8e68d19a5434e260c8dc283a533b771af5c3889ffdae8f2012e2e98d3881 WatchSource:0}: Error finding container 0fde8e68d19a5434e260c8dc283a533b771af5c3889ffdae8f2012e2e98d3881: Status 404 returned error can't find the container with id 0fde8e68d19a5434e260c8dc283a533b771af5c3889ffdae8f2012e2e98d3881 Oct 29 23:25:28.226863 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:28.226827 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 29 23:25:28.227311 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:28.226878 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" podUID="d5a70a55-4e5b-455a-b39f-e6e1b2489d03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 29 23:25:28.227681 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:28.227660 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" event={"ID":"045c37c8-57a0-4ed1-87a6-89f73eb079d2","Type":"ContainerStarted","Data":"e39d9fe266f398488311b1d5831b69fa8b47da0bac68ce2f76cd06d04b824ebb"} Oct 29 23:25:28.227733 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:28.227696 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" event={"ID":"045c37c8-57a0-4ed1-87a6-89f73eb079d2","Type":"ContainerStarted","Data":"0fde8e68d19a5434e260c8dc283a533b771af5c3889ffdae8f2012e2e98d3881"} Oct 29 23:25:28.267360 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:28.267302 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podStartSLOduration=1.2672874969999999 podStartE2EDuration="1.267287497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:28.265617385 +0000 UTC m=+697.628622127" watchObservedRunningTime="2026-10-29 23:25:28.267287497 +0000 UTC m=+697.630292219" Oct 29 23:25:28.543794 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:28.543693 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:29.222482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:29.214873 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 29 23:25:29.222482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:29.215027 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" podUID="d5a70a55-4e5b-455a-b39f-e6e1b2489d03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 29 23:25:30.221922 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:30.221878 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 29 23:25:30.222969 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:30.222542 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" podUID="d5a70a55-4e5b-455a-b39f-e6e1b2489d03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 29 23:25:31.230034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:31.229986 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:31.230723 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:31.230692 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:31.232711 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:31.232681 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:25:35.954433 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:35.954384 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:25:37.543579 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:37.543544 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:37.556065 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:37.556039 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:38.302177 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:38.302145 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:25:43.224134 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.224091 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr"] Oct 29 23:25:43.229218 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.229192 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.251787 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.251757 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-837245\"/\"default-dockercfg-gx7pm\"" Oct 29 23:25:43.251787 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.251782 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-837245\"/\"openshift-service-ca.crt\"" Oct 29 23:25:43.252003 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.251828 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-837245\"/\"premium-w6-837245-kserve-self-signed-certs\"" Oct 29 23:25:43.252003 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.251875 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-837245\"/\"kube-root-ca.crt\"" Oct 29 23:25:43.255399 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.255367 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr"] Oct 29 23:25:43.274211 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.274181 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-tmp-dir\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.274476 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.274441 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-model-cache\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.274618 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.274601 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-home\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.274760 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.274730 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmdwc\" (UniqueName: \"kubernetes.io/projected/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-kube-api-access-nmdwc\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.274937 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.274814 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-dshm\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.275024 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.274971 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-tls-certs\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.376044 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.376004 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-model-cache\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.376044 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.376047 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-home\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.376293 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.376072 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmdwc\" (UniqueName: \"kubernetes.io/projected/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-kube-api-access-nmdwc\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.376293 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.376120 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-dshm\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.376293 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.376152 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-tls-certs\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.376293 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.376227 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-tmp-dir\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.376529 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.376433 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-model-cache\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.376529 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.376506 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-home\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.376656 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.376640 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-tmp-dir\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.378471 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.378434 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-dshm\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.378655 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.378639 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-tls-certs\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.393727 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.393700 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmdwc\" (UniqueName: \"kubernetes.io/projected/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-kube-api-access-nmdwc\") pod \"premium-w6-837245-kserve-69bd764b57-c7vdr\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.544211 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.544125 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:43.847889 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:43.847807 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr"] Oct 29 23:25:43.851469 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:25:43.851422 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24364d6f_9a22_4b9c_b2ff_47bca2cbba80.slice/crio-25b90357f748056eaf69973176fd2dfa1fcefc48dd5576e9d255770896684796 WatchSource:0}: Error finding container 25b90357f748056eaf69973176fd2dfa1fcefc48dd5576e9d255770896684796: Status 404 returned error can't find the container with id 25b90357f748056eaf69973176fd2dfa1fcefc48dd5576e9d255770896684796 Oct 29 23:25:44.324481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:44.324421 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" event={"ID":"24364d6f-9a22-4b9c-b2ff-47bca2cbba80","Type":"ContainerStarted","Data":"1510a821cfcab096663700e38dae9f8deb17b5da76daa6ff2cb3bc4067d7936e"} Oct 29 23:25:44.324481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:44.324473 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" event={"ID":"24364d6f-9a22-4b9c-b2ff-47bca2cbba80","Type":"ContainerStarted","Data":"25b90357f748056eaf69973176fd2dfa1fcefc48dd5576e9d255770896684796"} Oct 29 23:25:44.359580 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:44.359529 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podStartSLOduration=1.359516597 podStartE2EDuration="1.359516597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:44.358561195 +0000 UTC m=+713.721565918" watchObservedRunningTime="2026-10-29 23:25:44.359516597 +0000 UTC m=+713.722521320" Oct 29 23:25:44.545258 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:44.545226 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:45.223800 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:25:45.223760 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:25:52.948312 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:52.948275 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w"] Oct 29 23:25:52.952586 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:52.952559 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:52.968632 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:52.968595 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9280e4\"/\"openshift-service-ca.crt\"" Oct 29 23:25:52.977638 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:52.977612 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9280e4\"/\"kube-root-ca.crt\"" Oct 29 23:25:52.977783 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:52.977694 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9280e4\"/\"default-dockercfg-gl7wv\"" Oct 29 23:25:52.978007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:52.977975 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9280e4\"/\"premium-w3-9280e4-kserve-self-signed-certs\"" Oct 29 23:25:52.980098 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:52.980078 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w"] Oct 29 23:25:53.119037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.118998 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-home\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.119037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.119039 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-dshm\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.119311 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.119072 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb49g\" (UniqueName: \"kubernetes.io/projected/f783e5d3-00f5-4110-af04-0dc0cabf30f5-kube-api-access-fb49g\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.119358 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.119336 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f783e5d3-00f5-4110-af04-0dc0cabf30f5-tls-certs\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.119394 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.119365 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-model-cache\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.119441 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.119409 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-tmp-dir\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.220535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.220424 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb49g\" (UniqueName: \"kubernetes.io/projected/f783e5d3-00f5-4110-af04-0dc0cabf30f5-kube-api-access-fb49g\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.220706 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.220587 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f783e5d3-00f5-4110-af04-0dc0cabf30f5-tls-certs\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.220706 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.220611 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-model-cache\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.220706 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.220648 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-tmp-dir\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.220706 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.220693 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-home\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.220933 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.220711 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-dshm\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.221100 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.221078 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-model-cache\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.221171 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.221143 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-home\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.221211 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.221177 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-tmp-dir\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.222975 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.222957 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-dshm\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.223165 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.223149 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f783e5d3-00f5-4110-af04-0dc0cabf30f5-tls-certs\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.237733 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.237707 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb49g\" (UniqueName: \"kubernetes.io/projected/f783e5d3-00f5-4110-af04-0dc0cabf30f5-kube-api-access-fb49g\") pod \"premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.265930 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.265894 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:53.544344 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.544261 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:53.557262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.557238 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:53.567486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:53.567436 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w"] Oct 29 23:25:53.571626 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:25:53.571601 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf783e5d3_00f5_4110_af04_0dc0cabf30f5.slice/crio-1685f8c773692bb59b8627889fed626f474697e7ad5a3a98e0c28928ce709265 WatchSource:0}: Error finding container 1685f8c773692bb59b8627889fed626f474697e7ad5a3a98e0c28928ce709265: Status 404 returned error can't find the container with id 1685f8c773692bb59b8627889fed626f474697e7ad5a3a98e0c28928ce709265 Oct 29 23:25:54.328277 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.328243 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn"] Oct 29 23:25:54.332269 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.332248 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.344075 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.344052 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-313d9c-openshift-default-dockercfg-bgdwt\"" Oct 29 23:25:54.362992 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.362963 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn"] Oct 29 23:25:54.388868 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.388832 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" event={"ID":"f783e5d3-00f5-4110-af04-0dc0cabf30f5","Type":"ContainerStarted","Data":"70645e011d51298d95c5be17b01181d0ef0e4d450bc1698c827661f0a3e93e6e"} Oct 29 23:25:54.388868 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.388873 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" event={"ID":"f783e5d3-00f5-4110-af04-0dc0cabf30f5","Type":"ContainerStarted","Data":"1685f8c773692bb59b8627889fed626f474697e7ad5a3a98e0c28928ce709265"} Oct 29 23:25:54.400601 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.400563 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:25:54.425364 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.425319 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podStartSLOduration=2.425306601 podStartE2EDuration="2.425306601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:54.424923975 +0000 UTC m=+723.787928697" watchObservedRunningTime="2026-10-29 23:25:54.425306601 +0000 UTC m=+723.788311324" Oct 29 23:25:54.434677 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.434645 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-data\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.434827 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.434717 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-workload-certs\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.434888 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.434818 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-token\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.434888 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.434850 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c272aaa1-73f3-4455-abe8-c61ca1047de3-istiod-ca-cert\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.434990 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.434936 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-workload-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.435056 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.434990 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-credential-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.435115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.435096 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qb89\" (UniqueName: \"kubernetes.io/projected/c272aaa1-73f3-4455-abe8-c61ca1047de3-kube-api-access-7qb89\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.435184 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.435172 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-envoy\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.435219 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.435205 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-podinfo\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.536197 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.536163 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-credential-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.536398 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.536257 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qb89\" (UniqueName: \"kubernetes.io/projected/c272aaa1-73f3-4455-abe8-c61ca1047de3-kube-api-access-7qb89\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.536398 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.536318 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-envoy\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.536398 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.536363 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-podinfo\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.536598 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.536410 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-data\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.536598 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.536502 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-workload-certs\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.536598 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.536543 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-token\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.536598 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.536578 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c272aaa1-73f3-4455-abe8-c61ca1047de3-istiod-ca-cert\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.536797 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.536610 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-credential-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.536797 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.536615 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-workload-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.536915 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.536795 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-data\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.536915 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.536902 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-workload-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.537762 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.537728 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-workload-certs\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.537883 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.537845 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c272aaa1-73f3-4455-abe8-c61ca1047de3-istiod-ca-cert\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.539137 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.539104 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-envoy\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.545183 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.545156 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-podinfo\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.553403 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.553375 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-token\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.553624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.553602 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qb89\" (UniqueName: \"kubernetes.io/projected/c272aaa1-73f3-4455-abe8-c61ca1047de3-kube-api-access-7qb89\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.646755 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.646625 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:54.973738 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.973704 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn"] Oct 29 23:25:54.977319 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:25:54.977285 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc272aaa1_73f3_4455_abe8_c61ca1047de3.slice/crio-e8e1496707e5d84866e0190e3eee6c14ce18a90f2ccdc7705c94e5be8cc6dd7d WatchSource:0}: Error finding container e8e1496707e5d84866e0190e3eee6c14ce18a90f2ccdc7705c94e5be8cc6dd7d: Status 404 returned error can't find the container with id e8e1496707e5d84866e0190e3eee6c14ce18a90f2ccdc7705c94e5be8cc6dd7d Oct 29 23:25:54.979799 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.979766 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:25:54.979869 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.979830 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:25:54.979869 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:54.979860 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:25:55.266349 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:55.266270 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:25:55.396275 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:55.396232 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" event={"ID":"c272aaa1-73f3-4455-abe8-c61ca1047de3","Type":"ContainerStarted","Data":"291a70375f0f73d7788e008e0ab608bc39055e9765bda3c96d26c656db9b4736"} Oct 29 23:25:55.396684 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:55.396300 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" event={"ID":"c272aaa1-73f3-4455-abe8-c61ca1047de3","Type":"ContainerStarted","Data":"e8e1496707e5d84866e0190e3eee6c14ce18a90f2ccdc7705c94e5be8cc6dd7d"} Oct 29 23:25:55.435170 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:55.435116 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" podStartSLOduration=1.435099442 podStartE2EDuration="1.435099442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:25:55.43270778 +0000 UTC m=+724.795712517" watchObservedRunningTime="2026-10-29 23:25:55.435099442 +0000 UTC m=+724.798104164" Oct 29 23:25:55.647710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:55.647626 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:25:56.649527 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:56.649490 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 29 23:25:56.649928 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:56.649549 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" podUID="c272aaa1-73f3-4455-abe8-c61ca1047de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 29 23:25:57.188927 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:57.188895 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:25:57.647148 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:57.647067 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 29 23:25:57.647148 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:57.647121 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" podUID="c272aaa1-73f3-4455-abe8-c61ca1047de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 29 23:25:58.649971 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:58.649928 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 29 23:25:58.650601 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:58.650561 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" podUID="c272aaa1-73f3-4455-abe8-c61ca1047de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 29 23:25:59.647177 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:59.647135 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 29 23:25:59.647373 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:25:59.647195 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" podUID="c272aaa1-73f3-4455-abe8-c61ca1047de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 29 23:26:00.647897 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:00.647800 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 29 23:26:00.647897 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:00.647872 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" podUID="c272aaa1-73f3-4455-abe8-c61ca1047de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 29 23:26:01.721588 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:01.721546 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:26:01.722104 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:01.721895 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:26:01.722873 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:01.722851 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:26:03.022836 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.022801 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8"] Oct 29 23:26:03.028052 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.028030 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" Oct 29 23:26:03.043696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.043675 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:26:03.052580 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.052554 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:26:03.052580 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.052554 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:26:03.052766 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.052595 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:26:03.057396 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.057359 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8"] Oct 29 23:26:03.177862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.177823 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbjmh\" (UniqueName: \"kubernetes.io/projected/2b6093b1-1a67-46ef-b324-c9433324fb9c-kube-api-access-hbjmh\") pod \"maas-api-revoke-sub-0df53e62-8t8g8\" (UID: \"2b6093b1-1a67-46ef-b324-c9433324fb9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" Oct 29 23:26:03.178053 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.178008 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6093b1-1a67-46ef-b324-c9433324fb9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0df53e62-8t8g8\" (UID: \"2b6093b1-1a67-46ef-b324-c9433324fb9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" Oct 29 23:26:03.178117 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.178076 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6093b1-1a67-46ef-b324-c9433324fb9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0df53e62-8t8g8\" (UID: \"2b6093b1-1a67-46ef-b324-c9433324fb9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" Oct 29 23:26:03.266229 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.266189 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:26:03.279408 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.279322 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:26:03.279609 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.279443 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbjmh\" (UniqueName: \"kubernetes.io/projected/2b6093b1-1a67-46ef-b324-c9433324fb9c-kube-api-access-hbjmh\") pod \"maas-api-revoke-sub-0df53e62-8t8g8\" (UID: \"2b6093b1-1a67-46ef-b324-c9433324fb9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" Oct 29 23:26:03.279677 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.279606 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6093b1-1a67-46ef-b324-c9433324fb9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0df53e62-8t8g8\" (UID: \"2b6093b1-1a67-46ef-b324-c9433324fb9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" Oct 29 23:26:03.279677 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.279661 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6093b1-1a67-46ef-b324-c9433324fb9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0df53e62-8t8g8\" (UID: \"2b6093b1-1a67-46ef-b324-c9433324fb9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" Oct 29 23:26:03.280231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.280212 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6093b1-1a67-46ef-b324-c9433324fb9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0df53e62-8t8g8\" (UID: \"2b6093b1-1a67-46ef-b324-c9433324fb9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" Oct 29 23:26:03.281979 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.281956 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6093b1-1a67-46ef-b324-c9433324fb9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0df53e62-8t8g8\" (UID: \"2b6093b1-1a67-46ef-b324-c9433324fb9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" Oct 29 23:26:03.297062 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.297024 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbjmh\" (UniqueName: \"kubernetes.io/projected/2b6093b1-1a67-46ef-b324-c9433324fb9c-kube-api-access-hbjmh\") pod \"maas-api-revoke-sub-0df53e62-8t8g8\" (UID: \"2b6093b1-1a67-46ef-b324-c9433324fb9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" Oct 29 23:26:03.342714 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.342665 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" Oct 29 23:26:03.490228 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.490194 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:26:03.647484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:03.647367 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8"] Oct 29 23:26:03.650943 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:26:03.650909 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6093b1_1a67_46ef_b324_c9433324fb9c.slice/crio-34df95c67f6cc8f6bc5dfc0e5563212b98ec8e6734f150d9ab57d857f2549cb7 WatchSource:0}: Error finding container 34df95c67f6cc8f6bc5dfc0e5563212b98ec8e6734f150d9ab57d857f2549cb7: Status 404 returned error can't find the container with id 34df95c67f6cc8f6bc5dfc0e5563212b98ec8e6734f150d9ab57d857f2549cb7 Oct 29 23:26:04.481300 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:04.481261 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" event={"ID":"2b6093b1-1a67-46ef-b324-c9433324fb9c","Type":"ContainerStarted","Data":"34df95c67f6cc8f6bc5dfc0e5563212b98ec8e6734f150d9ab57d857f2549cb7"} Oct 29 23:26:06.494493 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:06.494438 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" event={"ID":"2b6093b1-1a67-46ef-b324-c9433324fb9c","Type":"ContainerStarted","Data":"8b899ec98461060347fa08bcedfaea4b5145dbae8e54be5a847057c0025b078e"} Oct 29 23:26:06.529815 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:06.529751 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" podStartSLOduration=2.347226923 podStartE2EDuration="4.52973494s" totalImagesPullingTime="2.182508017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:06.526706505 +0000 UTC m=+735.889711210" watchObservedRunningTime="2026-10-29 23:26:06.52973494 +0000 UTC m=+735.892739664" Oct 29 23:26:07.501193 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:07.501158 6507 generic.go:356] "Generic (PLEG): container finished" podID="2b6093b1-1a67-46ef-b324-c9433324fb9c" containerID="8b899ec98461060347fa08bcedfaea4b5145dbae8e54be5a847057c0025b078e" exitCode=0 Oct 29 23:26:07.501616 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:07.501205 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" event={"ID":"2b6093b1-1a67-46ef-b324-c9433324fb9c","Type":"ContainerDied","Data":"8b899ec98461060347fa08bcedfaea4b5145dbae8e54be5a847057c0025b078e"} Oct 29 23:26:08.679560 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:08.679536 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" Oct 29 23:26:08.850210 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:08.850117 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b6093b1-1a67-46ef-b324-c9433324fb9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6093b1-1a67-46ef-b324-c9433324fb9c-maas-api-openshift-ca\") pod \"2b6093b1-1a67-46ef-b324-c9433324fb9c\" (UID: \"2b6093b1-1a67-46ef-b324-c9433324fb9c\") " Oct 29 23:26:08.850210 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:08.850185 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b6093b1-1a67-46ef-b324-c9433324fb9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6093b1-1a67-46ef-b324-c9433324fb9c-maas-api-xks-ca\") pod \"2b6093b1-1a67-46ef-b324-c9433324fb9c\" (UID: \"2b6093b1-1a67-46ef-b324-c9433324fb9c\") " Oct 29 23:26:08.850436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:08.850258 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6093b1-1a67-46ef-b324-c9433324fb9c-kube-api-access-hbjmh\" (UniqueName: \"kubernetes.io/projected/2b6093b1-1a67-46ef-b324-c9433324fb9c-kube-api-access-hbjmh\") pod \"2b6093b1-1a67-46ef-b324-c9433324fb9c\" (UID: \"2b6093b1-1a67-46ef-b324-c9433324fb9c\") " Oct 29 23:26:08.850541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:08.850506 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6093b1-1a67-46ef-b324-c9433324fb9c-maas-api-openshift-ca" pod "2b6093b1-1a67-46ef-b324-c9433324fb9c" (UID: "2b6093b1-1a67-46ef-b324-c9433324fb9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:08.850864 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:08.850839 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6093b1-1a67-46ef-b324-c9433324fb9c-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:08.853198 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:08.853170 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6093b1-1a67-46ef-b324-c9433324fb9c-maas-api-xks-ca" pod "2b6093b1-1a67-46ef-b324-c9433324fb9c" (UID: "2b6093b1-1a67-46ef-b324-c9433324fb9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:08.853286 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:08.853192 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6093b1-1a67-46ef-b324-c9433324fb9c-kube-api-access-hbjmh" pod "2b6093b1-1a67-46ef-b324-c9433324fb9c" (UID: "2b6093b1-1a67-46ef-b324-c9433324fb9c"). InnerVolumeSpecName "kube-api-access-hbjmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:08.952694 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:08.952656 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6093b1-1a67-46ef-b324-c9433324fb9c-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:08.952694 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:08.952694 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbjmh\" (UniqueName: \"kubernetes.io/projected/2b6093b1-1a67-46ef-b324-c9433324fb9c-kube-api-access-hbjmh\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:09.542929 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:09.542893 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" event={"ID":"2b6093b1-1a67-46ef-b324-c9433324fb9c","Type":"ContainerDied","Data":"34df95c67f6cc8f6bc5dfc0e5563212b98ec8e6734f150d9ab57d857f2549cb7"} Oct 29 23:26:09.542929 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:09.542933 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8" Oct 29 23:26:09.543140 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:09.542940 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34df95c67f6cc8f6bc5dfc0e5563212b98ec8e6734f150d9ab57d857f2549cb7" Oct 29 23:26:15.223527 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:26:15.223488 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:26:18.005225 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:18.005187 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:26:19.757978 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.757939 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9"] Oct 29 23:26:19.758620 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.758595 6507 cpu_manager.go:414] "containerMap: removing container" podUID="2b6093b1-1a67-46ef-b324-c9433324fb9c" containerName="revoke-keys" Oct 29 23:26:19.758620 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.758619 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6093b1-1a67-46ef-b324-c9433324fb9c" containerName="revoke-keys" Oct 29 23:26:19.758789 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.758746 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6093b1-1a67-46ef-b324-c9433324fb9c" containerName="revoke-keys" Oct 29 23:26:19.763510 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.763494 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9" Oct 29 23:26:19.784921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.784882 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:26:19.785108 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.784955 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:26:19.785175 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.784968 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:26:19.785175 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.784995 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:26:19.787302 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.787280 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9"] Oct 29 23:26:19.791611 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.791589 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v589\" (UniqueName: \"kubernetes.io/projected/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-kube-api-access-2v589\") pod \"maas-api-revoke-sub-0989d7c4-sl5d9\" (UID: \"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9" Oct 29 23:26:19.791751 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.791669 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0989d7c4-sl5d9\" (UID: \"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9" Oct 29 23:26:19.791751 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.791729 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0989d7c4-sl5d9\" (UID: \"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9" Oct 29 23:26:19.892953 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.892911 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0989d7c4-sl5d9\" (UID: \"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9" Oct 29 23:26:19.893147 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.892964 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0989d7c4-sl5d9\" (UID: \"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9" Oct 29 23:26:19.893147 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.893138 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v589\" (UniqueName: \"kubernetes.io/projected/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-kube-api-access-2v589\") pod \"maas-api-revoke-sub-0989d7c4-sl5d9\" (UID: \"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9" Oct 29 23:26:19.893692 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.893666 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0989d7c4-sl5d9\" (UID: \"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9" Oct 29 23:26:19.895922 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.895896 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0989d7c4-sl5d9\" (UID: \"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9" Oct 29 23:26:19.910088 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:19.910068 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v589\" (UniqueName: \"kubernetes.io/projected/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-kube-api-access-2v589\") pod \"maas-api-revoke-sub-0989d7c4-sl5d9\" (UID: \"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9" Oct 29 23:26:20.076774 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:20.076692 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9" Oct 29 23:26:20.374751 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:20.374658 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9"] Oct 29 23:26:20.378866 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:26:20.378833 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41db8ac2_ac01_48ce_927e_51d6f6e1eaa3.slice/crio-a1e5702d0b04a68200ab1cc315f01eb6bb41dfa912e978db0b0f2de66b27ba7f WatchSource:0}: Error finding container a1e5702d0b04a68200ab1cc315f01eb6bb41dfa912e978db0b0f2de66b27ba7f: Status 404 returned error can't find the container with id a1e5702d0b04a68200ab1cc315f01eb6bb41dfa912e978db0b0f2de66b27ba7f Oct 29 23:26:20.618794 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:20.618760 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9" event={"ID":"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3","Type":"ContainerStarted","Data":"a1e5702d0b04a68200ab1cc315f01eb6bb41dfa912e978db0b0f2de66b27ba7f"} Oct 29 23:26:21.625910 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:21.625811 6507 generic.go:356] "Generic (PLEG): container finished" podID="41db8ac2-ac01-48ce-927e-51d6f6e1eaa3" containerID="be7faa19e4833cbe392ba2efe492a28354a939866d02a9cc3bef4ac58636cb5a" exitCode=0 Oct 29 23:26:21.626291 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:21.625909 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9" event={"ID":"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3","Type":"ContainerDied","Data":"be7faa19e4833cbe392ba2efe492a28354a939866d02a9cc3bef4ac58636cb5a"} Oct 29 23:26:22.773107 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:22.773083 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9" Oct 29 23:26:22.826185 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:22.826158 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-kube-api-access-2v589\" (UniqueName: \"kubernetes.io/projected/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-kube-api-access-2v589\") pod \"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3\" (UID: \"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3\") " Oct 29 23:26:22.826392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:22.826373 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-maas-api-openshift-ca\") pod \"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3\" (UID: \"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3\") " Oct 29 23:26:22.826502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:22.826415 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-maas-api-xks-ca\") pod \"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3\" (UID: \"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3\") " Oct 29 23:26:22.826837 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:22.826802 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-maas-api-openshift-ca" pod "41db8ac2-ac01-48ce-927e-51d6f6e1eaa3" (UID: "41db8ac2-ac01-48ce-927e-51d6f6e1eaa3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:22.827019 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:22.827000 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:22.828571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:22.828545 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-maas-api-xks-ca" pod "41db8ac2-ac01-48ce-927e-51d6f6e1eaa3" (UID: "41db8ac2-ac01-48ce-927e-51d6f6e1eaa3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:22.829022 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:22.828996 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-kube-api-access-2v589" pod "41db8ac2-ac01-48ce-927e-51d6f6e1eaa3" (UID: "41db8ac2-ac01-48ce-927e-51d6f6e1eaa3"). InnerVolumeSpecName "kube-api-access-2v589". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:22.928679 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:22.928592 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:22.928679 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:22.928626 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v589\" (UniqueName: \"kubernetes.io/projected/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3-kube-api-access-2v589\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:23.638862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:23.638826 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9" Oct 29 23:26:23.639052 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:23.638865 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9" event={"ID":"41db8ac2-ac01-48ce-927e-51d6f6e1eaa3","Type":"ContainerDied","Data":"a1e5702d0b04a68200ab1cc315f01eb6bb41dfa912e978db0b0f2de66b27ba7f"} Oct 29 23:26:23.639052 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:23.638921 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1e5702d0b04a68200ab1cc315f01eb6bb41dfa912e978db0b0f2de66b27ba7f" Oct 29 23:26:27.680482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.659512 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk"] Oct 29 23:26:27.680482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.660886 6507 cpu_manager.go:414] "containerMap: removing container" podUID="41db8ac2-ac01-48ce-927e-51d6f6e1eaa3" containerName="revoke-keys" Oct 29 23:26:27.680482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.660910 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41db8ac2-ac01-48ce-927e-51d6f6e1eaa3" containerName="revoke-keys" Oct 29 23:26:27.680482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.661209 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="41db8ac2-ac01-48ce-927e-51d6f6e1eaa3" containerName="revoke-keys" Oct 29 23:26:27.723483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.698604 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk" Oct 29 23:26:27.728031 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.727995 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk"] Oct 29 23:26:27.777546 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.777000 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:26:27.777823 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.777223 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:26:27.778926 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.777353 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:26:27.782475 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.779474 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:26:27.808131 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.808090 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7e9924e-fkfqk\" (UID: \"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk" Oct 29 23:26:27.808607 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.808581 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwqfk\" (UniqueName: \"kubernetes.io/projected/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-kube-api-access-mwqfk\") pod \"maas-api-revoke-sub-a7e9924e-fkfqk\" (UID: \"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk" Oct 29 23:26:27.829173 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.829135 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7e9924e-fkfqk\" (UID: \"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk" Oct 29 23:26:27.932552 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.931136 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwqfk\" (UniqueName: \"kubernetes.io/projected/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-kube-api-access-mwqfk\") pod \"maas-api-revoke-sub-a7e9924e-fkfqk\" (UID: \"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk" Oct 29 23:26:27.932552 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.931214 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7e9924e-fkfqk\" (UID: \"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk" Oct 29 23:26:27.932552 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.931346 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7e9924e-fkfqk\" (UID: \"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk" Oct 29 23:26:27.936163 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.936131 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7e9924e-fkfqk\" (UID: \"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk" Oct 29 23:26:27.951375 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.951340 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwqfk\" (UniqueName: \"kubernetes.io/projected/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-kube-api-access-mwqfk\") pod \"maas-api-revoke-sub-a7e9924e-fkfqk\" (UID: \"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk" Oct 29 23:26:27.965531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:27.965357 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7e9924e-fkfqk\" (UID: \"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk" Oct 29 23:26:28.098497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:28.098438 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk" Oct 29 23:26:28.405317 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:28.405281 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk"] Oct 29 23:26:28.411299 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:26:28.411263 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b9a5ae_c4b9_4eff_b421_9f6c2d2b7921.slice/crio-e4145089945422099a668676c98787dc4c0caa45a9135e538960e86d5e5126ff WatchSource:0}: Error finding container e4145089945422099a668676c98787dc4c0caa45a9135e538960e86d5e5126ff: Status 404 returned error can't find the container with id e4145089945422099a668676c98787dc4c0caa45a9135e538960e86d5e5126ff Oct 29 23:26:28.682567 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:28.682479 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk" event={"ID":"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921","Type":"ContainerStarted","Data":"e4145089945422099a668676c98787dc4c0caa45a9135e538960e86d5e5126ff"} Oct 29 23:26:29.690162 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:29.690131 6507 generic.go:356] "Generic (PLEG): container finished" podID="55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921" containerID="872f9a79210858801240386626f0be7ef4059ebde4cba354019228222b475021" exitCode=0 Oct 29 23:26:29.690568 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:29.690179 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk" event={"ID":"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921","Type":"ContainerDied","Data":"872f9a79210858801240386626f0be7ef4059ebde4cba354019228222b475021"} Oct 29 23:26:30.853323 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:30.853291 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk" Oct 29 23:26:30.950320 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:30.950222 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x"] Oct 29 23:26:30.950825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:30.950806 6507 cpu_manager.go:414] "containerMap: removing container" podUID="55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921" containerName="revoke-keys" Oct 29 23:26:30.950825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:30.950823 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921" containerName="revoke-keys" Oct 29 23:26:30.951002 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:30.950931 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921" containerName="revoke-keys" Oct 29 23:26:30.954182 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:30.954160 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x" Oct 29 23:26:30.971772 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:30.971744 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-maas-api-openshift-ca\") pod \"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921\" (UID: \"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921\") " Oct 29 23:26:30.971912 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:30.971835 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-maas-api-xks-ca\") pod \"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921\" (UID: \"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921\") " Oct 29 23:26:30.971978 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:30.971916 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-kube-api-access-mwqfk\" (UniqueName: \"kubernetes.io/projected/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-kube-api-access-mwqfk\") pod \"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921\" (UID: \"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921\") " Oct 29 23:26:30.972162 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:30.972131 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-maas-api-openshift-ca" pod "55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921" (UID: "55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:30.972720 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:30.972695 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:30.974746 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:30.974720 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-kube-api-access-mwqfk" pod "55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921" (UID: "55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921"). InnerVolumeSpecName "kube-api-access-mwqfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:30.976261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:30.976235 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-maas-api-xks-ca" pod "55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921" (UID: "55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:30.998556 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:30.998182 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x"] Oct 29 23:26:31.074420 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.074390 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3325f452-zxc9x\" (UID: \"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x" Oct 29 23:26:31.074619 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.074531 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-582t9\" (UniqueName: \"kubernetes.io/projected/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-kube-api-access-582t9\") pod \"maas-api-revoke-sub-3325f452-zxc9x\" (UID: \"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x" Oct 29 23:26:31.074619 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.074610 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3325f452-zxc9x\" (UID: \"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x" Oct 29 23:26:31.074790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.074769 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:31.074838 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.074796 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwqfk\" (UniqueName: \"kubernetes.io/projected/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921-kube-api-access-mwqfk\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:31.175511 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.175451 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3325f452-zxc9x\" (UID: \"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x" Oct 29 23:26:31.175702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.175559 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3325f452-zxc9x\" (UID: \"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x" Oct 29 23:26:31.175702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.175643 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-582t9\" (UniqueName: \"kubernetes.io/projected/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-kube-api-access-582t9\") pod \"maas-api-revoke-sub-3325f452-zxc9x\" (UID: \"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x" Oct 29 23:26:31.176152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.176125 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3325f452-zxc9x\" (UID: \"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x" Oct 29 23:26:31.177940 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.177915 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3325f452-zxc9x\" (UID: \"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x" Oct 29 23:26:31.199906 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.199873 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-582t9\" (UniqueName: \"kubernetes.io/projected/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-kube-api-access-582t9\") pod \"maas-api-revoke-sub-3325f452-zxc9x\" (UID: \"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x" Oct 29 23:26:31.272414 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.272338 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x" Oct 29 23:26:31.570085 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.569996 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x"] Oct 29 23:26:31.574142 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:26:31.574114 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ffe3bc1_f6ca_4ffd_a481_f3fcf5ee18b8.slice/crio-8219b5fc5aaecb3093a04981b4c88dfccec89bffceacbfedc73e6face430e9b1 WatchSource:0}: Error finding container 8219b5fc5aaecb3093a04981b4c88dfccec89bffceacbfedc73e6face430e9b1: Status 404 returned error can't find the container with id 8219b5fc5aaecb3093a04981b4c88dfccec89bffceacbfedc73e6face430e9b1 Oct 29 23:26:31.704156 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.704124 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk" Oct 29 23:26:31.704345 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.704154 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk" event={"ID":"55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921","Type":"ContainerDied","Data":"e4145089945422099a668676c98787dc4c0caa45a9135e538960e86d5e5126ff"} Oct 29 23:26:31.704345 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.704202 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4145089945422099a668676c98787dc4c0caa45a9135e538960e86d5e5126ff" Oct 29 23:26:31.705513 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:31.705485 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x" event={"ID":"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8","Type":"ContainerStarted","Data":"8219b5fc5aaecb3093a04981b4c88dfccec89bffceacbfedc73e6face430e9b1"} Oct 29 23:26:32.131214 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:32.131124 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds"] Oct 29 23:26:32.135135 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:32.135111 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds" Oct 29 23:26:32.156333 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:32.156308 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds"] Oct 29 23:26:32.288094 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:32.288057 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eef75b8-42b2-42b7-8950-f62cc548cea0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0df51531-6wvds\" (UID: \"7eef75b8-42b2-42b7-8950-f62cc548cea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds" Oct 29 23:26:32.288241 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:32.288116 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eef75b8-42b2-42b7-8950-f62cc548cea0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0df51531-6wvds\" (UID: \"7eef75b8-42b2-42b7-8950-f62cc548cea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds" Oct 29 23:26:32.288241 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:32.288179 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vdl9\" (UniqueName: \"kubernetes.io/projected/7eef75b8-42b2-42b7-8950-f62cc548cea0-kube-api-access-2vdl9\") pod \"maas-api-revoke-sub-0df51531-6wvds\" (UID: \"7eef75b8-42b2-42b7-8950-f62cc548cea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds" Oct 29 23:26:32.389172 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:32.389086 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eef75b8-42b2-42b7-8950-f62cc548cea0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0df51531-6wvds\" (UID: \"7eef75b8-42b2-42b7-8950-f62cc548cea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds" Oct 29 23:26:32.389172 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:32.389151 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eef75b8-42b2-42b7-8950-f62cc548cea0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0df51531-6wvds\" (UID: \"7eef75b8-42b2-42b7-8950-f62cc548cea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds" Oct 29 23:26:32.389172 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:32.389169 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vdl9\" (UniqueName: \"kubernetes.io/projected/7eef75b8-42b2-42b7-8950-f62cc548cea0-kube-api-access-2vdl9\") pod \"maas-api-revoke-sub-0df51531-6wvds\" (UID: \"7eef75b8-42b2-42b7-8950-f62cc548cea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds" Oct 29 23:26:32.389888 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:32.389858 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eef75b8-42b2-42b7-8950-f62cc548cea0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0df51531-6wvds\" (UID: \"7eef75b8-42b2-42b7-8950-f62cc548cea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds" Oct 29 23:26:32.391504 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:32.391481 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eef75b8-42b2-42b7-8950-f62cc548cea0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0df51531-6wvds\" (UID: \"7eef75b8-42b2-42b7-8950-f62cc548cea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds" Oct 29 23:26:32.435181 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:32.431927 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vdl9\" (UniqueName: \"kubernetes.io/projected/7eef75b8-42b2-42b7-8950-f62cc548cea0-kube-api-access-2vdl9\") pod \"maas-api-revoke-sub-0df51531-6wvds\" (UID: \"7eef75b8-42b2-42b7-8950-f62cc548cea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds" Oct 29 23:26:32.449290 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:32.449260 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds" Oct 29 23:26:32.713211 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:32.713177 6507 generic.go:356] "Generic (PLEG): container finished" podID="6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8" containerID="504e990d320bdaaaebc7d5b102acb38a4062bcfcaf4e8139c0320f23d937e855" exitCode=0 Oct 29 23:26:32.713387 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:32.713238 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x" event={"ID":"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8","Type":"ContainerDied","Data":"504e990d320bdaaaebc7d5b102acb38a4062bcfcaf4e8139c0320f23d937e855"} Oct 29 23:26:32.758822 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:32.758787 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds"] Oct 29 23:26:32.767125 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:26:32.767086 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eef75b8_42b2_42b7_8950_f62cc548cea0.slice/crio-d1a32cf8c4eaadfa6e97ba4f61aa80c73bfff239ec328ba7432962dccf875b7c WatchSource:0}: Error finding container d1a32cf8c4eaadfa6e97ba4f61aa80c73bfff239ec328ba7432962dccf875b7c: Status 404 returned error can't find the container with id d1a32cf8c4eaadfa6e97ba4f61aa80c73bfff239ec328ba7432962dccf875b7c Oct 29 23:26:33.720525 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:33.720492 6507 generic.go:356] "Generic (PLEG): container finished" podID="7eef75b8-42b2-42b7-8950-f62cc548cea0" containerID="d388a055e2959ad9d669cad55b5e2849cf7296ef936dec531d6e1e1c35770c0b" exitCode=0 Oct 29 23:26:33.720909 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:33.720582 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds" event={"ID":"7eef75b8-42b2-42b7-8950-f62cc548cea0","Type":"ContainerDied","Data":"d388a055e2959ad9d669cad55b5e2849cf7296ef936dec531d6e1e1c35770c0b"} Oct 29 23:26:33.720909 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:33.720634 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds" event={"ID":"7eef75b8-42b2-42b7-8950-f62cc548cea0","Type":"ContainerStarted","Data":"d1a32cf8c4eaadfa6e97ba4f61aa80c73bfff239ec328ba7432962dccf875b7c"} Oct 29 23:26:33.877912 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:33.877884 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x" Oct 29 23:26:34.024924 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:34.024880 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-maas-api-openshift-ca\") pod \"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8\" (UID: \"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8\") " Oct 29 23:26:34.025121 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:34.024949 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-kube-api-access-582t9\" (UniqueName: \"kubernetes.io/projected/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-kube-api-access-582t9\") pod \"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8\" (UID: \"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8\") " Oct 29 23:26:34.025183 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:34.025148 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-maas-api-xks-ca\") pod \"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8\" (UID: \"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8\") " Oct 29 23:26:34.025334 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:34.025308 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-maas-api-openshift-ca" pod "6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8" (UID: "6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:34.025595 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:34.025571 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:34.027181 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:34.027129 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-kube-api-access-582t9" pod "6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8" (UID: "6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8"). InnerVolumeSpecName "kube-api-access-582t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:34.027181 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:34.027160 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-maas-api-xks-ca" pod "6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8" (UID: "6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:34.127095 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:34.127050 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:34.127095 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:34.127090 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-582t9\" (UniqueName: \"kubernetes.io/projected/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8-kube-api-access-582t9\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:34.727206 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:34.727171 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x" Oct 29 23:26:34.727673 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:34.727219 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x" event={"ID":"6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8","Type":"ContainerDied","Data":"8219b5fc5aaecb3093a04981b4c88dfccec89bffceacbfedc73e6face430e9b1"} Oct 29 23:26:34.727673 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:34.727293 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8219b5fc5aaecb3093a04981b4c88dfccec89bffceacbfedc73e6face430e9b1" Oct 29 23:26:34.877809 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:34.877781 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds" Oct 29 23:26:35.028949 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.028866 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs"] Oct 29 23:26:35.029413 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.029400 6507 cpu_manager.go:414] "containerMap: removing container" podUID="6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8" containerName="revoke-keys" Oct 29 23:26:35.029475 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.029414 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8" containerName="revoke-keys" Oct 29 23:26:35.029475 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.029435 6507 cpu_manager.go:414] "containerMap: removing container" podUID="7eef75b8-42b2-42b7-8950-f62cc548cea0" containerName="revoke-keys" Oct 29 23:26:35.029475 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.029440 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eef75b8-42b2-42b7-8950-f62cc548cea0" containerName="revoke-keys" Oct 29 23:26:35.029573 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.029560 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8" containerName="revoke-keys" Oct 29 23:26:35.029606 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.029574 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eef75b8-42b2-42b7-8950-f62cc548cea0" containerName="revoke-keys" Oct 29 23:26:35.032664 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.032647 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs" Oct 29 23:26:35.038306 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.038281 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eef75b8-42b2-42b7-8950-f62cc548cea0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eef75b8-42b2-42b7-8950-f62cc548cea0-maas-api-xks-ca\") pod \"7eef75b8-42b2-42b7-8950-f62cc548cea0\" (UID: \"7eef75b8-42b2-42b7-8950-f62cc548cea0\") " Oct 29 23:26:35.038426 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.038390 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eef75b8-42b2-42b7-8950-f62cc548cea0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eef75b8-42b2-42b7-8950-f62cc548cea0-maas-api-openshift-ca\") pod \"7eef75b8-42b2-42b7-8950-f62cc548cea0\" (UID: \"7eef75b8-42b2-42b7-8950-f62cc548cea0\") " Oct 29 23:26:35.038592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.038576 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eef75b8-42b2-42b7-8950-f62cc548cea0-kube-api-access-2vdl9\" (UniqueName: \"kubernetes.io/projected/7eef75b8-42b2-42b7-8950-f62cc548cea0-kube-api-access-2vdl9\") pod \"7eef75b8-42b2-42b7-8950-f62cc548cea0\" (UID: \"7eef75b8-42b2-42b7-8950-f62cc548cea0\") " Oct 29 23:26:35.038809 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.038782 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eef75b8-42b2-42b7-8950-f62cc548cea0-maas-api-openshift-ca" pod "7eef75b8-42b2-42b7-8950-f62cc548cea0" (UID: "7eef75b8-42b2-42b7-8950-f62cc548cea0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:35.039007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.038989 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eef75b8-42b2-42b7-8950-f62cc548cea0-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:35.040719 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.040693 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eef75b8-42b2-42b7-8950-f62cc548cea0-maas-api-xks-ca" pod "7eef75b8-42b2-42b7-8950-f62cc548cea0" (UID: "7eef75b8-42b2-42b7-8950-f62cc548cea0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:35.040802 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.040756 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eef75b8-42b2-42b7-8950-f62cc548cea0-kube-api-access-2vdl9" pod "7eef75b8-42b2-42b7-8950-f62cc548cea0" (UID: "7eef75b8-42b2-42b7-8950-f62cc548cea0"). InnerVolumeSpecName "kube-api-access-2vdl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:35.054156 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.054065 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs"] Oct 29 23:26:35.139948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.139915 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0baf759f-868e-47bc-b515-ae615233f4bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dac35cd-c7bcs\" (UID: \"0baf759f-868e-47bc-b515-ae615233f4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs" Oct 29 23:26:35.140145 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.139966 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz6j9\" (UniqueName: \"kubernetes.io/projected/0baf759f-868e-47bc-b515-ae615233f4bf-kube-api-access-fz6j9\") pod \"maas-api-revoke-sub-9dac35cd-c7bcs\" (UID: \"0baf759f-868e-47bc-b515-ae615233f4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs" Oct 29 23:26:35.140145 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.140126 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0baf759f-868e-47bc-b515-ae615233f4bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dac35cd-c7bcs\" (UID: \"0baf759f-868e-47bc-b515-ae615233f4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs" Oct 29 23:26:35.140259 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.140208 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vdl9\" (UniqueName: \"kubernetes.io/projected/7eef75b8-42b2-42b7-8950-f62cc548cea0-kube-api-access-2vdl9\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:35.140303 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.140251 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eef75b8-42b2-42b7-8950-f62cc548cea0-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:35.241266 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.241222 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0baf759f-868e-47bc-b515-ae615233f4bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dac35cd-c7bcs\" (UID: \"0baf759f-868e-47bc-b515-ae615233f4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs" Oct 29 23:26:35.241446 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.241287 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz6j9\" (UniqueName: \"kubernetes.io/projected/0baf759f-868e-47bc-b515-ae615233f4bf-kube-api-access-fz6j9\") pod \"maas-api-revoke-sub-9dac35cd-c7bcs\" (UID: \"0baf759f-868e-47bc-b515-ae615233f4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs" Oct 29 23:26:35.241446 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.241409 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0baf759f-868e-47bc-b515-ae615233f4bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dac35cd-c7bcs\" (UID: \"0baf759f-868e-47bc-b515-ae615233f4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs" Oct 29 23:26:35.242023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.242001 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0baf759f-868e-47bc-b515-ae615233f4bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dac35cd-c7bcs\" (UID: \"0baf759f-868e-47bc-b515-ae615233f4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs" Oct 29 23:26:35.243735 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.243709 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0baf759f-868e-47bc-b515-ae615233f4bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dac35cd-c7bcs\" (UID: \"0baf759f-868e-47bc-b515-ae615233f4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs" Oct 29 23:26:35.258290 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.258264 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz6j9\" (UniqueName: \"kubernetes.io/projected/0baf759f-868e-47bc-b515-ae615233f4bf-kube-api-access-fz6j9\") pod \"maas-api-revoke-sub-9dac35cd-c7bcs\" (UID: \"0baf759f-868e-47bc-b515-ae615233f4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs" Oct 29 23:26:35.364262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.364177 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs" Oct 29 23:26:35.667034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.666937 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs"] Oct 29 23:26:35.670949 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:26:35.670920 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0baf759f_868e_47bc_b515_ae615233f4bf.slice/crio-e8933578da493a19d5dd2cfb58a2dcea6f51c3904568391a44d67b40f85b0d33 WatchSource:0}: Error finding container e8933578da493a19d5dd2cfb58a2dcea6f51c3904568391a44d67b40f85b0d33: Status 404 returned error can't find the container with id e8933578da493a19d5dd2cfb58a2dcea6f51c3904568391a44d67b40f85b0d33 Oct 29 23:26:35.733994 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.733962 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs" event={"ID":"0baf759f-868e-47bc-b515-ae615233f4bf","Type":"ContainerStarted","Data":"e8933578da493a19d5dd2cfb58a2dcea6f51c3904568391a44d67b40f85b0d33"} Oct 29 23:26:35.735271 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.735250 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds" event={"ID":"7eef75b8-42b2-42b7-8950-f62cc548cea0","Type":"ContainerDied","Data":"d1a32cf8c4eaadfa6e97ba4f61aa80c73bfff239ec328ba7432962dccf875b7c"} Oct 29 23:26:35.735327 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.735287 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds" Oct 29 23:26:35.735376 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:35.735290 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1a32cf8c4eaadfa6e97ba4f61aa80c73bfff239ec328ba7432962dccf875b7c" Oct 29 23:26:36.381557 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:36.381520 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:26:36.742669 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:36.742637 6507 generic.go:356] "Generic (PLEG): container finished" podID="0baf759f-868e-47bc-b515-ae615233f4bf" containerID="6f4be05590aa9dd09872aaaad9d8486badee0228758e3593dae79b0c5ab28f25" exitCode=0 Oct 29 23:26:36.743060 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:36.742725 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs" event={"ID":"0baf759f-868e-47bc-b515-ae615233f4bf","Type":"ContainerDied","Data":"6f4be05590aa9dd09872aaaad9d8486badee0228758e3593dae79b0c5ab28f25"} Oct 29 23:26:37.296218 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:37.295574 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm"] Oct 29 23:26:37.310997 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:37.310966 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" Oct 29 23:26:37.326816 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:37.324358 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm"] Oct 29 23:26:37.498479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:37.498181 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16329f0e-c15c-4841-946d-c5df1de04a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8401733a-mhbvm\" (UID: \"16329f0e-c15c-4841-946d-c5df1de04a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" Oct 29 23:26:37.498479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:37.498327 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7s6p\" (UniqueName: \"kubernetes.io/projected/16329f0e-c15c-4841-946d-c5df1de04a6b-kube-api-access-k7s6p\") pod \"maas-api-revoke-sub-8401733a-mhbvm\" (UID: \"16329f0e-c15c-4841-946d-c5df1de04a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" Oct 29 23:26:37.498479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:37.498413 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16329f0e-c15c-4841-946d-c5df1de04a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8401733a-mhbvm\" (UID: \"16329f0e-c15c-4841-946d-c5df1de04a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" Oct 29 23:26:37.600542 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:37.600340 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7s6p\" (UniqueName: \"kubernetes.io/projected/16329f0e-c15c-4841-946d-c5df1de04a6b-kube-api-access-k7s6p\") pod \"maas-api-revoke-sub-8401733a-mhbvm\" (UID: \"16329f0e-c15c-4841-946d-c5df1de04a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" Oct 29 23:26:37.609482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:37.600972 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16329f0e-c15c-4841-946d-c5df1de04a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8401733a-mhbvm\" (UID: \"16329f0e-c15c-4841-946d-c5df1de04a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" Oct 29 23:26:37.609482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:37.601618 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16329f0e-c15c-4841-946d-c5df1de04a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8401733a-mhbvm\" (UID: \"16329f0e-c15c-4841-946d-c5df1de04a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" Oct 29 23:26:37.609482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:37.602600 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16329f0e-c15c-4841-946d-c5df1de04a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8401733a-mhbvm\" (UID: \"16329f0e-c15c-4841-946d-c5df1de04a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" Oct 29 23:26:37.650483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:37.629395 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7s6p\" (UniqueName: \"kubernetes.io/projected/16329f0e-c15c-4841-946d-c5df1de04a6b-kube-api-access-k7s6p\") pod \"maas-api-revoke-sub-8401733a-mhbvm\" (UID: \"16329f0e-c15c-4841-946d-c5df1de04a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" Oct 29 23:26:37.650483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:37.629973 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16329f0e-c15c-4841-946d-c5df1de04a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8401733a-mhbvm\" (UID: \"16329f0e-c15c-4841-946d-c5df1de04a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" Oct 29 23:26:37.655517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:37.652700 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" Oct 29 23:26:38.434049 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:38.434001 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm"] Oct 29 23:26:38.546987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:38.546664 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs" Oct 29 23:26:38.688842 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:38.685909 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0baf759f-868e-47bc-b515-ae615233f4bf-kube-api-access-fz6j9\" (UniqueName: \"kubernetes.io/projected/0baf759f-868e-47bc-b515-ae615233f4bf-kube-api-access-fz6j9\") pod \"0baf759f-868e-47bc-b515-ae615233f4bf\" (UID: \"0baf759f-868e-47bc-b515-ae615233f4bf\") " Oct 29 23:26:38.688842 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:38.686145 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0baf759f-868e-47bc-b515-ae615233f4bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0baf759f-868e-47bc-b515-ae615233f4bf-maas-api-openshift-ca\") pod \"0baf759f-868e-47bc-b515-ae615233f4bf\" (UID: \"0baf759f-868e-47bc-b515-ae615233f4bf\") " Oct 29 23:26:38.688842 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:38.686242 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0baf759f-868e-47bc-b515-ae615233f4bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0baf759f-868e-47bc-b515-ae615233f4bf-maas-api-xks-ca\") pod \"0baf759f-868e-47bc-b515-ae615233f4bf\" (UID: \"0baf759f-868e-47bc-b515-ae615233f4bf\") " Oct 29 23:26:38.688842 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:38.688760 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0baf759f-868e-47bc-b515-ae615233f4bf-maas-api-openshift-ca" pod "0baf759f-868e-47bc-b515-ae615233f4bf" (UID: "0baf759f-868e-47bc-b515-ae615233f4bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:38.694965 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:38.692269 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0baf759f-868e-47bc-b515-ae615233f4bf-maas-api-xks-ca" pod "0baf759f-868e-47bc-b515-ae615233f4bf" (UID: "0baf759f-868e-47bc-b515-ae615233f4bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:38.694965 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:38.694509 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0baf759f-868e-47bc-b515-ae615233f4bf-kube-api-access-fz6j9" pod "0baf759f-868e-47bc-b515-ae615233f4bf" (UID: "0baf759f-868e-47bc-b515-ae615233f4bf"). InnerVolumeSpecName "kube-api-access-fz6j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:38.792484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:38.791449 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs" event={"ID":"0baf759f-868e-47bc-b515-ae615233f4bf","Type":"ContainerDied","Data":"e8933578da493a19d5dd2cfb58a2dcea6f51c3904568391a44d67b40f85b0d33"} Oct 29 23:26:38.792484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:38.791554 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8933578da493a19d5dd2cfb58a2dcea6f51c3904568391a44d67b40f85b0d33" Oct 29 23:26:38.792484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:38.791653 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs" Oct 29 23:26:38.797588 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:38.797557 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz6j9\" (UniqueName: \"kubernetes.io/projected/0baf759f-868e-47bc-b515-ae615233f4bf-kube-api-access-fz6j9\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:38.797588 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:38.797590 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0baf759f-868e-47bc-b515-ae615233f4bf-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:38.797827 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:38.797603 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0baf759f-868e-47bc-b515-ae615233f4bf-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:38.819483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:38.804422 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" event={"ID":"16329f0e-c15c-4841-946d-c5df1de04a6b","Type":"ContainerStarted","Data":"4acb1994c8ef077000c8dee046b941f1516c89e98ba90b75b68c7cf455a7dd05"} Oct 29 23:26:39.184484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.181186 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft"] Oct 29 23:26:39.184484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.182076 6507 cpu_manager.go:414] "containerMap: removing container" podUID="0baf759f-868e-47bc-b515-ae615233f4bf" containerName="revoke-keys" Oct 29 23:26:39.184484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.182101 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0baf759f-868e-47bc-b515-ae615233f4bf" containerName="revoke-keys" Oct 29 23:26:39.184484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.182270 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="0baf759f-868e-47bc-b515-ae615233f4bf" containerName="revoke-keys" Oct 29 23:26:39.188536 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.188084 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.276032 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.275935 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft"] Oct 29 23:26:39.365349 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.365301 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-envoy\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.365559 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.365407 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52zcl\" (UniqueName: \"kubernetes.io/projected/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-kube-api-access-52zcl\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.365559 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.365444 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-credential-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.365559 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.365511 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-data\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.365559 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.365542 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-token\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.365840 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.365574 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istiod-ca-cert\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.365840 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.365633 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-podinfo\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.365840 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.365678 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-workload-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.365840 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.365754 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-workload-certs\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.473531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.473098 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52zcl\" (UniqueName: \"kubernetes.io/projected/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-kube-api-access-52zcl\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.473531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.473154 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-credential-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.473531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.473213 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-data\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.473531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.473244 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-token\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.473531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.473279 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istiod-ca-cert\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.473531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.473343 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-podinfo\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.473531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.473390 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-workload-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.473531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.473496 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-workload-certs\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.474435 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.473578 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-envoy\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.475291 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.475253 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istiod-ca-cert\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.475600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.475574 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-workload-certs\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.477249 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.475849 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-credential-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.477249 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.476426 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-data\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.477720 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.477698 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-envoy\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.478124 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.478103 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-workload-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.492477 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.492393 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-podinfo\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.495203 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.495175 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52zcl\" (UniqueName: \"kubernetes.io/projected/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-kube-api-access-52zcl\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.495882 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.495845 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-token\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.567534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.567105 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:39.632233 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.632178 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m"] Oct 29 23:26:39.684492 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.682651 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m"] Oct 29 23:26:39.684492 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.682795 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.695878 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.695852 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-0b2ae9-openshift-default-dockercfg-pwcxh\"" Oct 29 23:26:39.790850 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.790111 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-podinfo\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.790850 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.790187 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-workload-certs\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.790850 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.790245 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-data\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.790850 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.790290 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-workload-socket\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.790850 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.790362 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-credential-socket\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.790850 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.790442 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnxrj\" (UniqueName: \"kubernetes.io/projected/2d90e6ac-e91f-438e-ac8d-e4d027016263-kube-api-access-gnxrj\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.790850 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.790520 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-token\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.790850 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.790556 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d90e6ac-e91f-438e-ac8d-e4d027016263-istiod-ca-cert\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.790850 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.790604 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-envoy\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.896517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.895561 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-workload-certs\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.896517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.895637 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-data\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.896517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.895680 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-workload-socket\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.896517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.895775 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-credential-socket\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.896517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.895851 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnxrj\" (UniqueName: \"kubernetes.io/projected/2d90e6ac-e91f-438e-ac8d-e4d027016263-kube-api-access-gnxrj\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.896517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.895906 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-token\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.896517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.895936 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d90e6ac-e91f-438e-ac8d-e4d027016263-istiod-ca-cert\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.896517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.895982 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-envoy\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.896517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.896031 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-podinfo\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.896517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.896233 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-data\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.896517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.896486 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-workload-certs\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.897226 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.897042 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-workload-socket\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.897289 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.897265 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-credential-socket\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.898115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.898078 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d90e6ac-e91f-438e-ac8d-e4d027016263-istiod-ca-cert\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.900138 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.900008 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-envoy\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.900297 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.900268 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" event={"ID":"16329f0e-c15c-4841-946d-c5df1de04a6b","Type":"ContainerStarted","Data":"d287bcfeef48953e9aa0f5ef32690e849c0c61143f951c145894514f46b193ad"} Oct 29 23:26:39.900368 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.900342 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-podinfo\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.996048 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.996010 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnxrj\" (UniqueName: \"kubernetes.io/projected/2d90e6ac-e91f-438e-ac8d-e4d027016263-kube-api-access-gnxrj\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:39.997120 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:39.996599 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-token\") pod \"e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:40.053481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.051999 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:40.096637 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.096490 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" podStartSLOduration=3.096466447 podStartE2EDuration="3.096466447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:40.058055779 +0000 UTC m=+769.421060505" watchObservedRunningTime="2026-10-29 23:26:40.096466447 +0000 UTC m=+769.459471163" Oct 29 23:26:40.137531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.137487 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft"] Oct 29 23:26:40.193231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.192751 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:26:40.193231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.192852 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:26:40.193231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.192897 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:26:40.557682 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.557657 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m"] Oct 29 23:26:40.560678 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:26:40.560632 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d90e6ac_e91f_438e_ac8d_e4d027016263.slice/crio-3593935cae80e5cee170e596e4f3e12cbd6b41880dc980e1900c2c64e3b27c24 WatchSource:0}: Error finding container 3593935cae80e5cee170e596e4f3e12cbd6b41880dc980e1900c2c64e3b27c24: Status 404 returned error can't find the container with id 3593935cae80e5cee170e596e4f3e12cbd6b41880dc980e1900c2c64e3b27c24 Oct 29 23:26:40.563433 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.563398 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:26:40.563537 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.563490 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:26:40.563537 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.563531 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:26:40.909326 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.909291 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" event={"ID":"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17","Type":"ContainerStarted","Data":"55e7a2a10d25f882f13d266d94d1f18ff7b1c7050d4137162f214c2d8db3b566"} Oct 29 23:26:40.909495 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.909349 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" event={"ID":"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17","Type":"ContainerStarted","Data":"944fa190e6eb11c3ddc6d4ca0474b1b9ec42956763ac896ddd5df48f24378bca"} Oct 29 23:26:40.910971 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.910943 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" event={"ID":"2d90e6ac-e91f-438e-ac8d-e4d027016263","Type":"ContainerStarted","Data":"5eeb9cfa30f268f40eec8f72e6f05e836c6ff7dd7e78ed0a268f7ae86068e9d1"} Oct 29 23:26:40.911103 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.910987 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" event={"ID":"2d90e6ac-e91f-438e-ac8d-e4d027016263","Type":"ContainerStarted","Data":"3593935cae80e5cee170e596e4f3e12cbd6b41880dc980e1900c2c64e3b27c24"} Oct 29 23:26:40.912345 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.912317 6507 generic.go:356] "Generic (PLEG): container finished" podID="16329f0e-c15c-4841-946d-c5df1de04a6b" containerID="d287bcfeef48953e9aa0f5ef32690e849c0c61143f951c145894514f46b193ad" exitCode=0 Oct 29 23:26:40.912483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.912383 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" event={"ID":"16329f0e-c15c-4841-946d-c5df1de04a6b","Type":"ContainerDied","Data":"d287bcfeef48953e9aa0f5ef32690e849c0c61143f951c145894514f46b193ad"} Oct 29 23:26:40.949725 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:40.949671 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" podStartSLOduration=1.9496557079999999 podStartE2EDuration="1.949655708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:40.945596133 +0000 UTC m=+770.308600855" watchObservedRunningTime="2026-10-29 23:26:40.949655708 +0000 UTC m=+770.312660430" Oct 29 23:26:41.019252 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:41.019190 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" podStartSLOduration=2.01917298 podStartE2EDuration="2.01917298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:41.017833772 +0000 UTC m=+770.380838494" watchObservedRunningTime="2026-10-29 23:26:41.01917298 +0000 UTC m=+770.382177703" Oct 29 23:26:41.060997 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:41.060964 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:41.062470 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:41.062425 6507 patch_prober.go:29] interesting pod/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 29 23:26:41.062582 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:41.062506 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" podUID="2d90e6ac-e91f-438e-ac8d-e4d027016263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 29 23:26:41.568930 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:41.568493 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:41.570111 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:41.570077 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 29 23:26:41.570315 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:41.570289 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" podUID="b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 29 23:26:42.063401 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.062985 6507 patch_prober.go:29] interesting pod/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 29 23:26:42.063401 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.063053 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" podUID="2d90e6ac-e91f-438e-ac8d-e4d027016263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 29 23:26:42.187629 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.187603 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" Oct 29 23:26:42.194622 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.194604 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16329f0e-c15c-4841-946d-c5df1de04a6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16329f0e-c15c-4841-946d-c5df1de04a6b-maas-api-xks-ca\") pod \"16329f0e-c15c-4841-946d-c5df1de04a6b\" (UID: \"16329f0e-c15c-4841-946d-c5df1de04a6b\") " Oct 29 23:26:42.194703 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.194632 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16329f0e-c15c-4841-946d-c5df1de04a6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16329f0e-c15c-4841-946d-c5df1de04a6b-maas-api-openshift-ca\") pod \"16329f0e-c15c-4841-946d-c5df1de04a6b\" (UID: \"16329f0e-c15c-4841-946d-c5df1de04a6b\") " Oct 29 23:26:42.194703 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.194677 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16329f0e-c15c-4841-946d-c5df1de04a6b-kube-api-access-k7s6p\" (UniqueName: \"kubernetes.io/projected/16329f0e-c15c-4841-946d-c5df1de04a6b-kube-api-access-k7s6p\") pod \"16329f0e-c15c-4841-946d-c5df1de04a6b\" (UID: \"16329f0e-c15c-4841-946d-c5df1de04a6b\") " Oct 29 23:26:42.195048 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.195024 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16329f0e-c15c-4841-946d-c5df1de04a6b-maas-api-openshift-ca" pod "16329f0e-c15c-4841-946d-c5df1de04a6b" (UID: "16329f0e-c15c-4841-946d-c5df1de04a6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:42.196834 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.196809 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16329f0e-c15c-4841-946d-c5df1de04a6b-kube-api-access-k7s6p" pod "16329f0e-c15c-4841-946d-c5df1de04a6b" (UID: "16329f0e-c15c-4841-946d-c5df1de04a6b"). InnerVolumeSpecName "kube-api-access-k7s6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:42.199035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.199011 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16329f0e-c15c-4841-946d-c5df1de04a6b-maas-api-xks-ca" pod "16329f0e-c15c-4841-946d-c5df1de04a6b" (UID: "16329f0e-c15c-4841-946d-c5df1de04a6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:42.296138 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.296098 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7s6p\" (UniqueName: \"kubernetes.io/projected/16329f0e-c15c-4841-946d-c5df1de04a6b-kube-api-access-k7s6p\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:42.296138 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.296140 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16329f0e-c15c-4841-946d-c5df1de04a6b-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:42.296350 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.296156 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16329f0e-c15c-4841-946d-c5df1de04a6b-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:42.568278 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.568196 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 29 23:26:42.568278 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.568255 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" podUID="b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 29 23:26:42.966649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.966551 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" event={"ID":"16329f0e-c15c-4841-946d-c5df1de04a6b","Type":"ContainerDied","Data":"4acb1994c8ef077000c8dee046b941f1516c89e98ba90b75b68c7cf455a7dd05"} Oct 29 23:26:42.966649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.966615 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4acb1994c8ef077000c8dee046b941f1516c89e98ba90b75b68c7cf455a7dd05" Oct 29 23:26:42.966649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:42.966568 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm" Oct 29 23:26:43.061387 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:43.061351 6507 patch_prober.go:29] interesting pod/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 29 23:26:43.061589 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:43.061407 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" podUID="2d90e6ac-e91f-438e-ac8d-e4d027016263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 29 23:26:43.568237 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:43.568199 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 29 23:26:43.568427 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:43.568256 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" podUID="b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 29 23:26:44.060959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.060919 6507 patch_prober.go:29] interesting pod/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 29 23:26:44.061444 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.061411 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" podUID="2d90e6ac-e91f-438e-ac8d-e4d027016263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 29 23:26:44.567877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.567841 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 29 23:26:44.568063 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.567892 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" podUID="b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 29 23:26:44.772234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.772204 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks"] Oct 29 23:26:44.773004 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.772972 6507 cpu_manager.go:414] "containerMap: removing container" podUID="16329f0e-c15c-4841-946d-c5df1de04a6b" containerName="revoke-keys" Oct 29 23:26:44.773123 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.773111 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16329f0e-c15c-4841-946d-c5df1de04a6b" containerName="revoke-keys" Oct 29 23:26:44.773282 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.773272 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="16329f0e-c15c-4841-946d-c5df1de04a6b" containerName="revoke-keys" Oct 29 23:26:44.776449 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.776432 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks" Oct 29 23:26:44.798956 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.798923 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:26:44.798956 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.798953 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:26:44.799181 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.799018 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:26:44.799181 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.798929 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:26:44.819561 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.815299 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks"] Oct 29 23:26:44.866993 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.866924 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kmwj\" (UniqueName: \"kubernetes.io/projected/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-kube-api-access-4kmwj\") pod \"maas-api-revoke-sub-e7c73a3f-l9zks\" (UID: \"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks" Oct 29 23:26:44.867202 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.867058 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7c73a3f-l9zks\" (UID: \"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks" Oct 29 23:26:44.867278 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.867232 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7c73a3f-l9zks\" (UID: \"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks" Oct 29 23:26:44.968598 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.968561 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7c73a3f-l9zks\" (UID: \"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks" Oct 29 23:26:44.968807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.968626 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kmwj\" (UniqueName: \"kubernetes.io/projected/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-kube-api-access-4kmwj\") pod \"maas-api-revoke-sub-e7c73a3f-l9zks\" (UID: \"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks" Oct 29 23:26:44.968807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.968671 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7c73a3f-l9zks\" (UID: \"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks" Oct 29 23:26:44.969409 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.969383 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7c73a3f-l9zks\" (UID: \"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks" Oct 29 23:26:44.971375 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.971356 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7c73a3f-l9zks\" (UID: \"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks" Oct 29 23:26:44.986212 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:44.986181 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kmwj\" (UniqueName: \"kubernetes.io/projected/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-kube-api-access-4kmwj\") pod \"maas-api-revoke-sub-e7c73a3f-l9zks\" (UID: \"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks" Oct 29 23:26:45.061030 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:45.060971 6507 patch_prober.go:29] interesting pod/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 29 23:26:45.061437 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:45.061060 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" podUID="2d90e6ac-e91f-438e-ac8d-e4d027016263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 29 23:26:45.093312 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:45.093235 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks" Oct 29 23:26:45.291590 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:26:45.291553 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:26:45.470473 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:45.470036 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks"] Oct 29 23:26:45.568315 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:45.568279 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 29 23:26:45.568527 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:45.568341 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" podUID="b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 29 23:26:46.065949 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.065900 6507 patch_prober.go:29] interesting pod/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 29 23:26:46.074498 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.070583 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" podUID="2d90e6ac-e91f-438e-ac8d-e4d027016263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 29 23:26:46.074689 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.074625 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf"] Oct 29 23:26:46.083168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.083134 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks" event={"ID":"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca","Type":"ContainerStarted","Data":"8597f005d920a192812457c5ae7ac9d3ac15c8b2f0cf2735578686963fc2be15"} Oct 29 23:26:46.083318 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.083178 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf"] Oct 29 23:26:46.083318 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.083243 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf" Oct 29 23:26:46.188834 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.188807 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7803083-cb6b-4c2c-91a0-8ed401b2e794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54ef1cd5-65nmf\" (UID: \"c7803083-cb6b-4c2c-91a0-8ed401b2e794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf" Oct 29 23:26:46.189000 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.188842 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7803083-cb6b-4c2c-91a0-8ed401b2e794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54ef1cd5-65nmf\" (UID: \"c7803083-cb6b-4c2c-91a0-8ed401b2e794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf" Oct 29 23:26:46.189000 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.188888 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr2jp\" (UniqueName: \"kubernetes.io/projected/c7803083-cb6b-4c2c-91a0-8ed401b2e794-kube-api-access-dr2jp\") pod \"maas-api-revoke-sub-54ef1cd5-65nmf\" (UID: \"c7803083-cb6b-4c2c-91a0-8ed401b2e794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf" Oct 29 23:26:46.290061 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.290023 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7803083-cb6b-4c2c-91a0-8ed401b2e794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54ef1cd5-65nmf\" (UID: \"c7803083-cb6b-4c2c-91a0-8ed401b2e794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf" Oct 29 23:26:46.290061 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.290067 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7803083-cb6b-4c2c-91a0-8ed401b2e794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54ef1cd5-65nmf\" (UID: \"c7803083-cb6b-4c2c-91a0-8ed401b2e794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf" Oct 29 23:26:46.290259 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.290118 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2jp\" (UniqueName: \"kubernetes.io/projected/c7803083-cb6b-4c2c-91a0-8ed401b2e794-kube-api-access-dr2jp\") pod \"maas-api-revoke-sub-54ef1cd5-65nmf\" (UID: \"c7803083-cb6b-4c2c-91a0-8ed401b2e794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf" Oct 29 23:26:46.291075 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.291052 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7803083-cb6b-4c2c-91a0-8ed401b2e794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54ef1cd5-65nmf\" (UID: \"c7803083-cb6b-4c2c-91a0-8ed401b2e794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf" Oct 29 23:26:46.295410 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.295391 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7803083-cb6b-4c2c-91a0-8ed401b2e794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54ef1cd5-65nmf\" (UID: \"c7803083-cb6b-4c2c-91a0-8ed401b2e794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf" Oct 29 23:26:46.347482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.336347 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2jp\" (UniqueName: \"kubernetes.io/projected/c7803083-cb6b-4c2c-91a0-8ed401b2e794-kube-api-access-dr2jp\") pod \"maas-api-revoke-sub-54ef1cd5-65nmf\" (UID: \"c7803083-cb6b-4c2c-91a0-8ed401b2e794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf" Oct 29 23:26:46.397860 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.397829 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf" Oct 29 23:26:46.569491 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.569416 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 29 23:26:46.570118 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.569708 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" podUID="b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 29 23:26:46.801643 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:46.801553 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf"] Oct 29 23:26:46.826478 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:26:46.824743 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7803083_cb6b_4c2c_91a0_8ed401b2e794.slice/crio-8fa299eff498fc6864c43afbff4fb45e668dc671ed60fe24eac2babee01a4393 WatchSource:0}: Error finding container 8fa299eff498fc6864c43afbff4fb45e668dc671ed60fe24eac2babee01a4393: Status 404 returned error can't find the container with id 8fa299eff498fc6864c43afbff4fb45e668dc671ed60fe24eac2babee01a4393 Oct 29 23:26:47.061538 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:47.061435 6507 patch_prober.go:29] interesting pod/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 29 23:26:47.061538 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:47.061515 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" podUID="2d90e6ac-e91f-438e-ac8d-e4d027016263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 29 23:26:47.087203 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:47.087168 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf" event={"ID":"c7803083-cb6b-4c2c-91a0-8ed401b2e794","Type":"ContainerStarted","Data":"8fa299eff498fc6864c43afbff4fb45e668dc671ed60fe24eac2babee01a4393"} Oct 29 23:26:47.089022 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:47.088985 6507 generic.go:356] "Generic (PLEG): container finished" podID="208de026-0c8d-4e9f-a6dc-6fdcc394c4ca" containerID="8a2dbef67968f68c3c77270765bc12ed56e9adb5b5bec321ea46dab8a9f8bce4" exitCode=0 Oct 29 23:26:47.089139 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:47.089039 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks" event={"ID":"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca","Type":"ContainerDied","Data":"8a2dbef67968f68c3c77270765bc12ed56e9adb5b5bec321ea46dab8a9f8bce4"} Oct 29 23:26:47.583806 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:47.583774 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:47.585748 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:47.585718 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:47.593747 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:47.593716 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:26:48.153499 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:48.152828 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:48.154142 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:48.154119 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:48.158619 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:48.158451 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:26:48.161744 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:48.161717 6507 generic.go:356] "Generic (PLEG): container finished" podID="c7803083-cb6b-4c2c-91a0-8ed401b2e794" containerID="17a2c7627d599f637ec67437c291c995bf2a0e4387f7ac02f95a8110f42e7e8c" exitCode=0 Oct 29 23:26:48.161867 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:48.161813 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf" event={"ID":"c7803083-cb6b-4c2c-91a0-8ed401b2e794","Type":"ContainerDied","Data":"17a2c7627d599f637ec67437c291c995bf2a0e4387f7ac02f95a8110f42e7e8c"} Oct 29 23:26:48.337185 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:48.337154 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks" Oct 29 23:26:48.517078 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:48.517046 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-maas-api-xks-ca\") pod \"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca\" (UID: \"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca\") " Oct 29 23:26:48.517279 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:48.517109 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-maas-api-openshift-ca\") pod \"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca\" (UID: \"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca\") " Oct 29 23:26:48.517345 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:48.517286 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-kube-api-access-4kmwj\" (UniqueName: \"kubernetes.io/projected/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-kube-api-access-4kmwj\") pod \"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca\" (UID: \"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca\") " Oct 29 23:26:48.517470 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:48.517434 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-maas-api-openshift-ca" pod "208de026-0c8d-4e9f-a6dc-6fdcc394c4ca" (UID: "208de026-0c8d-4e9f-a6dc-6fdcc394c4ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:48.517645 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:48.517629 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:48.519214 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:48.519189 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-maas-api-xks-ca" pod "208de026-0c8d-4e9f-a6dc-6fdcc394c4ca" (UID: "208de026-0c8d-4e9f-a6dc-6fdcc394c4ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:48.519446 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:48.519428 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-kube-api-access-4kmwj" pod "208de026-0c8d-4e9f-a6dc-6fdcc394c4ca" (UID: "208de026-0c8d-4e9f-a6dc-6fdcc394c4ca"). InnerVolumeSpecName "kube-api-access-4kmwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:48.619053 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:48.619016 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kmwj\" (UniqueName: \"kubernetes.io/projected/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-kube-api-access-4kmwj\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:48.619053 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:48.619051 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:49.206605 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:49.199614 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks" event={"ID":"208de026-0c8d-4e9f-a6dc-6fdcc394c4ca","Type":"ContainerDied","Data":"8597f005d920a192812457c5ae7ac9d3ac15c8b2f0cf2735578686963fc2be15"} Oct 29 23:26:49.206605 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:49.199673 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8597f005d920a192812457c5ae7ac9d3ac15c8b2f0cf2735578686963fc2be15" Oct 29 23:26:49.206605 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:49.199752 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks" Oct 29 23:26:49.358483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:49.358434 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf" Oct 29 23:26:49.427644 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:49.427601 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7803083-cb6b-4c2c-91a0-8ed401b2e794-kube-api-access-dr2jp\" (UniqueName: \"kubernetes.io/projected/c7803083-cb6b-4c2c-91a0-8ed401b2e794-kube-api-access-dr2jp\") pod \"c7803083-cb6b-4c2c-91a0-8ed401b2e794\" (UID: \"c7803083-cb6b-4c2c-91a0-8ed401b2e794\") " Oct 29 23:26:49.427818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:49.427691 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7803083-cb6b-4c2c-91a0-8ed401b2e794-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7803083-cb6b-4c2c-91a0-8ed401b2e794-maas-api-openshift-ca\") pod \"c7803083-cb6b-4c2c-91a0-8ed401b2e794\" (UID: \"c7803083-cb6b-4c2c-91a0-8ed401b2e794\") " Oct 29 23:26:49.427870 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:49.427859 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7803083-cb6b-4c2c-91a0-8ed401b2e794-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7803083-cb6b-4c2c-91a0-8ed401b2e794-maas-api-xks-ca\") pod \"c7803083-cb6b-4c2c-91a0-8ed401b2e794\" (UID: \"c7803083-cb6b-4c2c-91a0-8ed401b2e794\") " Oct 29 23:26:49.428075 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:49.428044 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7803083-cb6b-4c2c-91a0-8ed401b2e794-maas-api-openshift-ca" pod "c7803083-cb6b-4c2c-91a0-8ed401b2e794" (UID: "c7803083-cb6b-4c2c-91a0-8ed401b2e794"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:49.428340 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:49.428318 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7803083-cb6b-4c2c-91a0-8ed401b2e794-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:49.429982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:49.429952 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7803083-cb6b-4c2c-91a0-8ed401b2e794-kube-api-access-dr2jp" pod "c7803083-cb6b-4c2c-91a0-8ed401b2e794" (UID: "c7803083-cb6b-4c2c-91a0-8ed401b2e794"). InnerVolumeSpecName "kube-api-access-dr2jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:49.430339 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:49.430320 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7803083-cb6b-4c2c-91a0-8ed401b2e794-maas-api-xks-ca" pod "c7803083-cb6b-4c2c-91a0-8ed401b2e794" (UID: "c7803083-cb6b-4c2c-91a0-8ed401b2e794"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:49.529064 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:49.528976 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7803083-cb6b-4c2c-91a0-8ed401b2e794-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:49.529064 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:49.529007 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr2jp\" (UniqueName: \"kubernetes.io/projected/c7803083-cb6b-4c2c-91a0-8ed401b2e794-kube-api-access-dr2jp\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:50.206606 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:50.206578 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf" Oct 29 23:26:50.206761 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:50.206620 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf" event={"ID":"c7803083-cb6b-4c2c-91a0-8ed401b2e794","Type":"ContainerDied","Data":"8fa299eff498fc6864c43afbff4fb45e668dc671ed60fe24eac2babee01a4393"} Oct 29 23:26:50.206761 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:50.206667 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fa299eff498fc6864c43afbff4fb45e668dc671ed60fe24eac2babee01a4393" Oct 29 23:26:51.625586 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.625552 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2"] Oct 29 23:26:51.626065 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.626052 6507 cpu_manager.go:414] "containerMap: removing container" podUID="208de026-0c8d-4e9f-a6dc-6fdcc394c4ca" containerName="revoke-keys" Oct 29 23:26:51.626108 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.626065 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208de026-0c8d-4e9f-a6dc-6fdcc394c4ca" containerName="revoke-keys" Oct 29 23:26:51.626108 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.626090 6507 cpu_manager.go:414] "containerMap: removing container" podUID="c7803083-cb6b-4c2c-91a0-8ed401b2e794" containerName="revoke-keys" Oct 29 23:26:51.626108 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.626095 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7803083-cb6b-4c2c-91a0-8ed401b2e794" containerName="revoke-keys" Oct 29 23:26:51.626196 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.626180 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7803083-cb6b-4c2c-91a0-8ed401b2e794" containerName="revoke-keys" Oct 29 23:26:51.626196 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.626192 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="208de026-0c8d-4e9f-a6dc-6fdcc394c4ca" containerName="revoke-keys" Oct 29 23:26:51.630671 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.630646 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2" Oct 29 23:26:51.642527 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.642505 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:26:51.647454 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.647424 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwt6t\" (UniqueName: \"kubernetes.io/projected/423fb383-714f-478d-b772-0d7678789327-kube-api-access-mwt6t\") pod \"maas-api-revoke-sub-18d78c27-h5pt2\" (UID: \"423fb383-714f-478d-b772-0d7678789327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2" Oct 29 23:26:51.647666 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.647647 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423fb383-714f-478d-b772-0d7678789327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18d78c27-h5pt2\" (UID: \"423fb383-714f-478d-b772-0d7678789327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2" Oct 29 23:26:51.647711 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.647682 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423fb383-714f-478d-b772-0d7678789327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18d78c27-h5pt2\" (UID: \"423fb383-714f-478d-b772-0d7678789327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2" Oct 29 23:26:51.652001 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.651969 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:26:51.652110 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.652000 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:26:51.652110 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.652000 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:26:51.655045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.655026 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2"] Oct 29 23:26:51.748829 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.748774 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423fb383-714f-478d-b772-0d7678789327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18d78c27-h5pt2\" (UID: \"423fb383-714f-478d-b772-0d7678789327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2" Oct 29 23:26:51.748829 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.748834 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423fb383-714f-478d-b772-0d7678789327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18d78c27-h5pt2\" (UID: \"423fb383-714f-478d-b772-0d7678789327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2" Oct 29 23:26:51.749084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.748921 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwt6t\" (UniqueName: \"kubernetes.io/projected/423fb383-714f-478d-b772-0d7678789327-kube-api-access-mwt6t\") pod \"maas-api-revoke-sub-18d78c27-h5pt2\" (UID: \"423fb383-714f-478d-b772-0d7678789327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2" Oct 29 23:26:51.749437 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.749417 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423fb383-714f-478d-b772-0d7678789327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18d78c27-h5pt2\" (UID: \"423fb383-714f-478d-b772-0d7678789327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2" Oct 29 23:26:51.751253 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.751228 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423fb383-714f-478d-b772-0d7678789327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18d78c27-h5pt2\" (UID: \"423fb383-714f-478d-b772-0d7678789327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2" Oct 29 23:26:51.765517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.765490 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwt6t\" (UniqueName: \"kubernetes.io/projected/423fb383-714f-478d-b772-0d7678789327-kube-api-access-mwt6t\") pod \"maas-api-revoke-sub-18d78c27-h5pt2\" (UID: \"423fb383-714f-478d-b772-0d7678789327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2" Oct 29 23:26:51.943582 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:51.943552 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2" Oct 29 23:26:52.244551 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:52.244526 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2"] Oct 29 23:26:52.247437 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:26:52.247402 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423fb383_714f_478d_b772_0d7678789327.slice/crio-49463c52cdd217a4367ada0deb02a3c7269226cddcc0d8b6afe0b153d8765457 WatchSource:0}: Error finding container 49463c52cdd217a4367ada0deb02a3c7269226cddcc0d8b6afe0b153d8765457: Status 404 returned error can't find the container with id 49463c52cdd217a4367ada0deb02a3c7269226cddcc0d8b6afe0b153d8765457 Oct 29 23:26:53.245712 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:53.245674 6507 generic.go:356] "Generic (PLEG): container finished" podID="423fb383-714f-478d-b772-0d7678789327" containerID="9882f38ea9565ce8f9817aad3ea54a678a5f6e613c75a1a488ba461a8d2782f5" exitCode=0 Oct 29 23:26:53.246798 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:53.246645 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2" event={"ID":"423fb383-714f-478d-b772-0d7678789327","Type":"ContainerDied","Data":"9882f38ea9565ce8f9817aad3ea54a678a5f6e613c75a1a488ba461a8d2782f5"} Oct 29 23:26:53.247007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:53.246992 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2" event={"ID":"423fb383-714f-478d-b772-0d7678789327","Type":"ContainerStarted","Data":"49463c52cdd217a4367ada0deb02a3c7269226cddcc0d8b6afe0b153d8765457"} Oct 29 23:26:54.557496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.552577 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:26:54.557496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.553854 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:26:54.574610 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.574565 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc"] Oct 29 23:26:54.723710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.669848 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.723710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.677430 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc"] Oct 29 23:26:54.870552 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.870185 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-credential-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.870552 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.870260 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-istio-envoy\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.870552 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.870349 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-istio-data\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.870552 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.870411 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914b9029-3376-4fad-addf-ce1b73391b5e-istiod-ca-cert\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.870552 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.870441 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgwrd\" (UniqueName: \"kubernetes.io/projected/914b9029-3376-4fad-addf-ce1b73391b5e-kube-api-access-tgwrd\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.871487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.871039 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914b9029-3376-4fad-addf-ce1b73391b5e-istio-podinfo\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.871487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.871147 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914b9029-3376-4fad-addf-ce1b73391b5e-istio-token\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.871487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.871211 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-workload-certs\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.871487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.871263 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-workload-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.968181 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.954711 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:26:54.977723 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.977680 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-istio-data\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.977986 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.977965 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914b9029-3376-4fad-addf-ce1b73391b5e-istiod-ca-cert\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.978124 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.978107 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgwrd\" (UniqueName: \"kubernetes.io/projected/914b9029-3376-4fad-addf-ce1b73391b5e-kube-api-access-tgwrd\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.978254 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.978241 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914b9029-3376-4fad-addf-ce1b73391b5e-istio-podinfo\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.978430 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.978412 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914b9029-3376-4fad-addf-ce1b73391b5e-istio-token\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.978647 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.978629 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-workload-certs\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.978819 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.978801 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-workload-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.978999 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.978977 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-credential-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.979118 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.979102 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-istio-envoy\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.979946 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.979922 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-istio-data\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.988476 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.980684 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914b9029-3376-4fad-addf-ce1b73391b5e-istiod-ca-cert\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:54.990788 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.989022 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-workload-certs\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:55.035275 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.989266 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-workload-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:55.035543 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:54.989762 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-credential-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:55.052679 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:55.052644 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914b9029-3376-4fad-addf-ce1b73391b5e-istio-podinfo\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:55.070483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:55.055982 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914b9029-3376-4fad-addf-ce1b73391b5e-istio-token\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:55.073043 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:55.072972 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgwrd\" (UniqueName: \"kubernetes.io/projected/914b9029-3376-4fad-addf-ce1b73391b5e-kube-api-access-tgwrd\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:55.090921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:55.090849 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-istio-envoy\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:55.234482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:55.224267 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:55.253945 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:55.234795 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 29 23:26:55.253945 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:55.249581 6507 patch_prober.go:29] interesting pod/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:26:55.253945 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:55.249649 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" podUID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:26:55.306050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:55.305996 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:26:55.419131 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:55.419075 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:26:55.561729 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:55.561617 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:26:55.567690 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:55.567653 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:26:55.972934 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:55.955751 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:26:55.972934 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:55.958896 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:26:56.250534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.250405 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 29 23:26:56.252637 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.252605 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:26:56.445446 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.445390 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:26:56.697480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.541330 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2" Oct 29 23:26:56.697480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.567536 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 29 23:26:56.697480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.567588 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 23:26:56.697480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.672305 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2" event={"ID":"423fb383-714f-478d-b772-0d7678789327","Type":"ContainerDied","Data":"49463c52cdd217a4367ada0deb02a3c7269226cddcc0d8b6afe0b153d8765457"} Oct 29 23:26:56.697480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.697315 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49463c52cdd217a4367ada0deb02a3c7269226cddcc0d8b6afe0b153d8765457" Oct 29 23:26:56.697480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.697365 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:26:56.697480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.697402 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:26:56.743519 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.736216 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:26:56.750692 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.744548 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:26:56.796875 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.796849 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:26:56.797044 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.796896 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:26:56.797044 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.796920 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:26:56.884982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.884949 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423fb383-714f-478d-b772-0d7678789327-kube-api-access-mwt6t\" (UniqueName: \"kubernetes.io/projected/423fb383-714f-478d-b772-0d7678789327-kube-api-access-mwt6t\") pod \"423fb383-714f-478d-b772-0d7678789327\" (UID: \"423fb383-714f-478d-b772-0d7678789327\") " Oct 29 23:26:56.884982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.884990 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/423fb383-714f-478d-b772-0d7678789327-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423fb383-714f-478d-b772-0d7678789327-maas-api-openshift-ca\") pod \"423fb383-714f-478d-b772-0d7678789327\" (UID: \"423fb383-714f-478d-b772-0d7678789327\") " Oct 29 23:26:56.885200 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.885125 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/423fb383-714f-478d-b772-0d7678789327-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423fb383-714f-478d-b772-0d7678789327-maas-api-xks-ca\") pod \"423fb383-714f-478d-b772-0d7678789327\" (UID: \"423fb383-714f-478d-b772-0d7678789327\") " Oct 29 23:26:56.885467 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.885424 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423fb383-714f-478d-b772-0d7678789327-maas-api-openshift-ca" pod "423fb383-714f-478d-b772-0d7678789327" (UID: "423fb383-714f-478d-b772-0d7678789327"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:56.885651 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.885622 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423fb383-714f-478d-b772-0d7678789327-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:56.887238 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.887205 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423fb383-714f-478d-b772-0d7678789327-maas-api-xks-ca" pod "423fb383-714f-478d-b772-0d7678789327" (UID: "423fb383-714f-478d-b772-0d7678789327"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:56.887772 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.887750 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423fb383-714f-478d-b772-0d7678789327-kube-api-access-mwt6t" pod "423fb383-714f-478d-b772-0d7678789327" (UID: "423fb383-714f-478d-b772-0d7678789327"). InnerVolumeSpecName "kube-api-access-mwt6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:56.937485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.936888 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc"] Oct 29 23:26:56.978301 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.978259 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:26:56.978542 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.978513 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:26:56.978720 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.978697 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:26:56.986573 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.986532 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423fb383-714f-478d-b772-0d7678789327-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:56.986573 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:56.986562 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwt6t\" (UniqueName: \"kubernetes.io/projected/423fb383-714f-478d-b772-0d7678789327-kube-api-access-mwt6t\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:26:57.680209 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:57.680169 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" event={"ID":"914b9029-3376-4fad-addf-ce1b73391b5e","Type":"ContainerStarted","Data":"c1c31919fa712acb0dbfc994634efe6f143fbe9caaba34b744c7a47db4519fb0"} Oct 29 23:26:57.680400 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:57.680227 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" event={"ID":"914b9029-3376-4fad-addf-ce1b73391b5e","Type":"ContainerStarted","Data":"07556dc31cff91821d323abb1e56833c8d1e0e4f3eaee538c4682f53706ff01d"} Oct 29 23:26:57.680752 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:57.680723 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2" Oct 29 23:26:57.720686 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:57.720603 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" podStartSLOduration=3.720584178 podStartE2EDuration="3.720584178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:57.719704895 +0000 UTC m=+787.082709618" watchObservedRunningTime="2026-10-29 23:26:57.720584178 +0000 UTC m=+787.083588902" Oct 29 23:26:58.237169 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.237125 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:26:58.238782 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.238755 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 29 23:26:58.238904 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.238807 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" podUID="914b9029-3376-4fad-addf-ce1b73391b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 29 23:26:58.531648 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.528312 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j"] Oct 29 23:26:58.531648 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.529199 6507 cpu_manager.go:414] "containerMap: removing container" podUID="423fb383-714f-478d-b772-0d7678789327" containerName="revoke-keys" Oct 29 23:26:58.531648 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.529221 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423fb383-714f-478d-b772-0d7678789327" containerName="revoke-keys" Oct 29 23:26:58.531648 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.529404 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="423fb383-714f-478d-b772-0d7678789327" containerName="revoke-keys" Oct 29 23:26:58.535315 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.535287 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" Oct 29 23:26:58.552737 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.552707 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j"] Oct 29 23:26:58.570203 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.569799 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:26:58.570203 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.569876 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:26:58.572793 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.572769 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:26:58.572793 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.572792 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:26:58.632515 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.632446 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9scq6\" (UniqueName: \"kubernetes.io/projected/07008149-7cf0-44aa-a624-c2bd45af32f8-kube-api-access-9scq6\") pod \"maas-api-revoke-sub-9e5a66bd-9f78j\" (UID: \"07008149-7cf0-44aa-a624-c2bd45af32f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" Oct 29 23:26:58.632696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.632648 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07008149-7cf0-44aa-a624-c2bd45af32f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e5a66bd-9f78j\" (UID: \"07008149-7cf0-44aa-a624-c2bd45af32f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" Oct 29 23:26:58.632696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.632685 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07008149-7cf0-44aa-a624-c2bd45af32f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e5a66bd-9f78j\" (UID: \"07008149-7cf0-44aa-a624-c2bd45af32f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" Oct 29 23:26:58.733910 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.733871 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07008149-7cf0-44aa-a624-c2bd45af32f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e5a66bd-9f78j\" (UID: \"07008149-7cf0-44aa-a624-c2bd45af32f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" Oct 29 23:26:58.734386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.733922 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07008149-7cf0-44aa-a624-c2bd45af32f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e5a66bd-9f78j\" (UID: \"07008149-7cf0-44aa-a624-c2bd45af32f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" Oct 29 23:26:58.734386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.733975 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9scq6\" (UniqueName: \"kubernetes.io/projected/07008149-7cf0-44aa-a624-c2bd45af32f8-kube-api-access-9scq6\") pod \"maas-api-revoke-sub-9e5a66bd-9f78j\" (UID: \"07008149-7cf0-44aa-a624-c2bd45af32f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" Oct 29 23:26:58.734785 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.734763 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07008149-7cf0-44aa-a624-c2bd45af32f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e5a66bd-9f78j\" (UID: \"07008149-7cf0-44aa-a624-c2bd45af32f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" Oct 29 23:26:58.736386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.736363 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07008149-7cf0-44aa-a624-c2bd45af32f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e5a66bd-9f78j\" (UID: \"07008149-7cf0-44aa-a624-c2bd45af32f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" Oct 29 23:26:58.753551 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.753516 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9scq6\" (UniqueName: \"kubernetes.io/projected/07008149-7cf0-44aa-a624-c2bd45af32f8-kube-api-access-9scq6\") pod \"maas-api-revoke-sub-9e5a66bd-9f78j\" (UID: \"07008149-7cf0-44aa-a624-c2bd45af32f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" Oct 29 23:26:58.849141 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:58.849058 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" Oct 29 23:26:59.240527 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:59.240345 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j"] Oct 29 23:26:59.241044 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:59.241014 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 29 23:26:59.241393 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:59.241363 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" podUID="914b9029-3376-4fad-addf-ce1b73391b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 29 23:26:59.245101 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:26:59.245078 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07008149_7cf0_44aa_a624_c2bd45af32f8.slice/crio-be31c9f2955c0cff7528deadd22745eab8b9c49d60950fd856295abf1cf6b850 WatchSource:0}: Error finding container be31c9f2955c0cff7528deadd22745eab8b9c49d60950fd856295abf1cf6b850: Status 404 returned error can't find the container with id be31c9f2955c0cff7528deadd22745eab8b9c49d60950fd856295abf1cf6b850 Oct 29 23:26:59.694644 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:59.694608 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" event={"ID":"07008149-7cf0-44aa-a624-c2bd45af32f8","Type":"ContainerStarted","Data":"7153e2f79f052ff3e47104c5e10ef47b4edd211c982c74be87f95d0a36983d2f"} Oct 29 23:26:59.694842 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:59.694665 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" event={"ID":"07008149-7cf0-44aa-a624-c2bd45af32f8","Type":"ContainerStarted","Data":"be31c9f2955c0cff7528deadd22745eab8b9c49d60950fd856295abf1cf6b850"} Oct 29 23:26:59.753566 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:26:59.753516 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" podStartSLOduration=1.75350176 podStartE2EDuration="1.75350176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:59.750758066 +0000 UTC m=+789.113762789" watchObservedRunningTime="2026-10-29 23:26:59.75350176 +0000 UTC m=+789.116506482" Oct 29 23:27:00.237142 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:00.237097 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 29 23:27:00.237308 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:00.237151 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" podUID="914b9029-3376-4fad-addf-ce1b73391b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 29 23:27:00.723804 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:00.723248 6507 generic.go:356] "Generic (PLEG): container finished" podID="07008149-7cf0-44aa-a624-c2bd45af32f8" containerID="7153e2f79f052ff3e47104c5e10ef47b4edd211c982c74be87f95d0a36983d2f" exitCode=0 Oct 29 23:27:00.723804 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:00.723406 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" event={"ID":"07008149-7cf0-44aa-a624-c2bd45af32f8","Type":"ContainerDied","Data":"7153e2f79f052ff3e47104c5e10ef47b4edd211c982c74be87f95d0a36983d2f"} Oct 29 23:27:01.237334 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:01.237301 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 29 23:27:01.237763 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:01.237351 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" podUID="914b9029-3376-4fad-addf-ce1b73391b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 29 23:27:01.897634 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:01.897603 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" Oct 29 23:27:01.972951 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:01.972916 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07008149-7cf0-44aa-a624-c2bd45af32f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07008149-7cf0-44aa-a624-c2bd45af32f8-maas-api-xks-ca\") pod \"07008149-7cf0-44aa-a624-c2bd45af32f8\" (UID: \"07008149-7cf0-44aa-a624-c2bd45af32f8\") " Oct 29 23:27:01.973096 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:01.972959 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07008149-7cf0-44aa-a624-c2bd45af32f8-kube-api-access-9scq6\" (UniqueName: \"kubernetes.io/projected/07008149-7cf0-44aa-a624-c2bd45af32f8-kube-api-access-9scq6\") pod \"07008149-7cf0-44aa-a624-c2bd45af32f8\" (UID: \"07008149-7cf0-44aa-a624-c2bd45af32f8\") " Oct 29 23:27:01.973096 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:01.973042 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07008149-7cf0-44aa-a624-c2bd45af32f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07008149-7cf0-44aa-a624-c2bd45af32f8-maas-api-openshift-ca\") pod \"07008149-7cf0-44aa-a624-c2bd45af32f8\" (UID: \"07008149-7cf0-44aa-a624-c2bd45af32f8\") " Oct 29 23:27:01.973516 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:01.973486 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07008149-7cf0-44aa-a624-c2bd45af32f8-maas-api-openshift-ca" pod "07008149-7cf0-44aa-a624-c2bd45af32f8" (UID: "07008149-7cf0-44aa-a624-c2bd45af32f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:01.975335 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:01.975302 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07008149-7cf0-44aa-a624-c2bd45af32f8-kube-api-access-9scq6" pod "07008149-7cf0-44aa-a624-c2bd45af32f8" (UID: "07008149-7cf0-44aa-a624-c2bd45af32f8"). InnerVolumeSpecName "kube-api-access-9scq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:01.975438 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:01.975387 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07008149-7cf0-44aa-a624-c2bd45af32f8-maas-api-xks-ca" pod "07008149-7cf0-44aa-a624-c2bd45af32f8" (UID: "07008149-7cf0-44aa-a624-c2bd45af32f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:27:02.074695 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:02.074648 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07008149-7cf0-44aa-a624-c2bd45af32f8-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:27:02.074695 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:02.074689 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9scq6\" (UniqueName: \"kubernetes.io/projected/07008149-7cf0-44aa-a624-c2bd45af32f8-kube-api-access-9scq6\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:27:02.074695 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:02.074702 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07008149-7cf0-44aa-a624-c2bd45af32f8-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:27:02.237251 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:02.237164 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 29 23:27:02.237251 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:02.237239 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" podUID="914b9029-3376-4fad-addf-ce1b73391b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 29 23:27:02.742351 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:02.742316 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" Oct 29 23:27:02.742570 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:02.742343 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j" event={"ID":"07008149-7cf0-44aa-a624-c2bd45af32f8","Type":"ContainerDied","Data":"be31c9f2955c0cff7528deadd22745eab8b9c49d60950fd856295abf1cf6b850"} Oct 29 23:27:02.742570 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:02.742405 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be31c9f2955c0cff7528deadd22745eab8b9c49d60950fd856295abf1cf6b850" Oct 29 23:27:03.236602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:03.236560 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 29 23:27:03.236771 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:03.236622 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" podUID="914b9029-3376-4fad-addf-ce1b73391b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 29 23:27:04.497621 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:04.497581 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:27:04.498700 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:04.498667 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:27:04.499895 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:04.499860 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:27:08.292110 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:08.292068 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:27:08.879930 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:08.879888 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j"] Oct 29 23:27:08.880448 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:08.880435 6507 cpu_manager.go:414] "containerMap: removing container" podUID="07008149-7cf0-44aa-a624-c2bd45af32f8" containerName="revoke-keys" Oct 29 23:27:08.880554 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:08.880449 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07008149-7cf0-44aa-a624-c2bd45af32f8" containerName="revoke-keys" Oct 29 23:27:08.880589 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:08.880571 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="07008149-7cf0-44aa-a624-c2bd45af32f8" containerName="revoke-keys" Oct 29 23:27:08.884258 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:08.884235 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j" Oct 29 23:27:08.896440 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:08.896414 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:27:08.905779 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:08.905742 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:27:08.905779 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:08.905742 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:27:08.906029 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:08.905782 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:27:08.909809 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:08.909779 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j"] Oct 29 23:27:08.999861 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:08.996398 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdc4af09-9qm7j\" (UID: \"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j" Oct 29 23:27:08.999861 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:08.996476 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-987lh\" (UniqueName: \"kubernetes.io/projected/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-kube-api-access-987lh\") pod \"maas-api-revoke-sub-bdc4af09-9qm7j\" (UID: \"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j" Oct 29 23:27:08.999861 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:08.996541 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdc4af09-9qm7j\" (UID: \"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j" Oct 29 23:27:09.099982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.099949 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdc4af09-9qm7j\" (UID: \"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j" Oct 29 23:27:09.100272 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.100242 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-987lh\" (UniqueName: \"kubernetes.io/projected/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-kube-api-access-987lh\") pod \"maas-api-revoke-sub-bdc4af09-9qm7j\" (UID: \"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j" Oct 29 23:27:09.100448 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.100405 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdc4af09-9qm7j\" (UID: \"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j" Oct 29 23:27:09.101695 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.101658 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdc4af09-9qm7j\" (UID: \"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j" Oct 29 23:27:09.124991 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.124959 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdc4af09-9qm7j\" (UID: \"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j" Oct 29 23:27:09.150294 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.150222 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-987lh\" (UniqueName: \"kubernetes.io/projected/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-kube-api-access-987lh\") pod \"maas-api-revoke-sub-bdc4af09-9qm7j\" (UID: \"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j" Oct 29 23:27:09.206567 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.206526 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j" Oct 29 23:27:09.624484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.623485 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j"] Oct 29 23:27:09.744079 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.744024 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6"] Oct 29 23:27:09.757475 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.756699 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.782322 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.782294 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6"] Oct 29 23:27:09.827430 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.826772 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e3a005-5ced-4440-814c-f2713230bd95-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.827430 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.826864 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e3a005-5ced-4440-814c-f2713230bd95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.827430 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.826896 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e3a005-5ced-4440-814c-f2713230bd95-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.827430 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.826926 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e3a005-5ced-4440-814c-f2713230bd95-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.827430 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.827000 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e3a005-5ced-4440-814c-f2713230bd95-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.827430 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.827090 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz62h\" (UniqueName: \"kubernetes.io/projected/b9e3a005-5ced-4440-814c-f2713230bd95-kube-api-access-jz62h\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.827430 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.827132 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e3a005-5ced-4440-814c-f2713230bd95-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.827430 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.827184 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e3a005-5ced-4440-814c-f2713230bd95-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.827430 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.827221 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e3a005-5ced-4440-814c-f2713230bd95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.851632 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.851595 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j" event={"ID":"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469","Type":"ContainerStarted","Data":"d3996b5a33c98bc5e95b7076eb7842d3f4a00a113d4d51313664a630cc8a6068"} Oct 29 23:27:09.928643 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.928608 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e3a005-5ced-4440-814c-f2713230bd95-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.928825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.928669 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e3a005-5ced-4440-814c-f2713230bd95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.928825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.928698 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e3a005-5ced-4440-814c-f2713230bd95-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.928825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.928724 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e3a005-5ced-4440-814c-f2713230bd95-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.928825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.928786 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e3a005-5ced-4440-814c-f2713230bd95-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.929057 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.928865 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz62h\" (UniqueName: \"kubernetes.io/projected/b9e3a005-5ced-4440-814c-f2713230bd95-kube-api-access-jz62h\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.929057 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.928907 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e3a005-5ced-4440-814c-f2713230bd95-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.929057 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.928960 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e3a005-5ced-4440-814c-f2713230bd95-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.929057 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.928997 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e3a005-5ced-4440-814c-f2713230bd95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.929264 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.929240 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e3a005-5ced-4440-814c-f2713230bd95-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.929327 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.929293 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e3a005-5ced-4440-814c-f2713230bd95-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.930651 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.930605 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e3a005-5ced-4440-814c-f2713230bd95-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.930917 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.930870 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e3a005-5ced-4440-814c-f2713230bd95-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.931019 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.930981 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e3a005-5ced-4440-814c-f2713230bd95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.933056 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.933028 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e3a005-5ced-4440-814c-f2713230bd95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.933170 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.933058 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e3a005-5ced-4440-814c-f2713230bd95-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.946197 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.946152 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e3a005-5ced-4440-814c-f2713230bd95-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:09.947088 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:09.947049 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz62h\" (UniqueName: \"kubernetes.io/projected/b9e3a005-5ced-4440-814c-f2713230bd95-kube-api-access-jz62h\") pod \"maas-default-gateway-openshift-default-746ff8f49f-8gxl6\" (UID: \"b9e3a005-5ced-4440-814c-f2713230bd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:10.088602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.088559 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:10.415389 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.415346 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6"] Oct 29 23:27:10.437545 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.428974 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k"] Oct 29 23:27:10.437545 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.431411 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:27:10.437545 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.431526 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:27:10.437545 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.431572 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:27:10.437545 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.435346 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.453126 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.453090 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2670a5\"/\"openshift-service-ca.crt\"" Oct 29 23:27:10.454774 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.454750 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2670a5\"/\"distinct-w0-2670a5-kserve-self-signed-certs\"" Oct 29 23:27:10.456940 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.456916 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k"] Oct 29 23:27:10.462608 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.462582 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2670a5\"/\"kube-root-ca.crt\"" Oct 29 23:27:10.462608 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.462594 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2670a5\"/\"default-dockercfg-dxlxd\"" Oct 29 23:27:10.536544 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.536515 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-model-cache\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.536729 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.536604 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-dshm\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.536821 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.536796 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87843995-cc71-42f6-ac3d-a5f3b13907c0-tls-certs\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.536904 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.536885 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-home\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.536970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.536918 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v424c\" (UniqueName: \"kubernetes.io/projected/87843995-cc71-42f6-ac3d-a5f3b13907c0-kube-api-access-v424c\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.536970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.536938 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-tmp-dir\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.562421 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:27:10.562376 6507 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde2ebe4_9d4a_4b3f_b684_ed6dc850a469.slice/crio-f9693e0ef38e60c797c61b7c840199c9e6ade5a290bacfd9d7aaeee5906d5ee3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde2ebe4_9d4a_4b3f_b684_ed6dc850a469.slice/crio-conmon-f9693e0ef38e60c797c61b7c840199c9e6ade5a290bacfd9d7aaeee5906d5ee3.scope\": RecentStats: unable to find data in memory cache]" Oct 29 23:27:10.650482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.644339 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87843995-cc71-42f6-ac3d-a5f3b13907c0-tls-certs\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.650482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.644433 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-home\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.650482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.644479 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v424c\" (UniqueName: \"kubernetes.io/projected/87843995-cc71-42f6-ac3d-a5f3b13907c0-kube-api-access-v424c\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.650482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.644503 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-tmp-dir\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.650482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.644592 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-model-cache\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.650482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.644636 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-dshm\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.650482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.646138 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-tmp-dir\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.650482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.646510 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-model-cache\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.650482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.648091 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-home\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.650482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.650282 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87843995-cc71-42f6-ac3d-a5f3b13907c0-tls-certs\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.651478 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.651223 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-dshm\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.670302 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.670217 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v424c\" (UniqueName: \"kubernetes.io/projected/87843995-cc71-42f6-ac3d-a5f3b13907c0-kube-api-access-v424c\") pod \"distinct-w0-2670a5-kserve-579bf9ff4d-sv95k\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.751858 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.751823 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:10.861284 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.861225 6507 generic.go:356] "Generic (PLEG): container finished" podID="cde2ebe4-9d4a-4b3f-b684-ed6dc850a469" containerID="f9693e0ef38e60c797c61b7c840199c9e6ade5a290bacfd9d7aaeee5906d5ee3" exitCode=0 Oct 29 23:27:10.861521 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.861327 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j" event={"ID":"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469","Type":"ContainerDied","Data":"f9693e0ef38e60c797c61b7c840199c9e6ade5a290bacfd9d7aaeee5906d5ee3"} Oct 29 23:27:10.864052 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.864012 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" event={"ID":"b9e3a005-5ced-4440-814c-f2713230bd95","Type":"ContainerStarted","Data":"2ef38398f3937abe29aea61417df389fe26cf8f2cdbff5e701fe5bd14af14c6a"} Oct 29 23:27:10.864210 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.864078 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" event={"ID":"b9e3a005-5ced-4440-814c-f2713230bd95","Type":"ContainerStarted","Data":"0b2c44f5760cb75251c2d9948cec3ce0582fc7ba8393bab5bfb7a01b6f036795"} Oct 29 23:27:10.938982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:10.938913 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" podStartSLOduration=1.938894905 podStartE2EDuration="1.938894905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:10.935654249 +0000 UTC m=+800.298658985" watchObservedRunningTime="2026-10-29 23:27:10.938894905 +0000 UTC m=+800.301899629" Oct 29 23:27:11.100622 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:11.100583 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-8gxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 29 23:27:11.100892 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:11.100857 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" podUID="b9e3a005-5ced-4440-814c-f2713230bd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 29 23:27:11.101092 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:11.101074 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:11.114308 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:11.114235 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k"] Oct 29 23:27:11.872198 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:11.872159 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" event={"ID":"87843995-cc71-42f6-ac3d-a5f3b13907c0","Type":"ContainerStarted","Data":"ce3a8f265d66811d4ed0daf1a57a6b541eeca3d513c2129053f8c267860b34fa"} Oct 29 23:27:11.872721 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:11.872207 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" event={"ID":"87843995-cc71-42f6-ac3d-a5f3b13907c0","Type":"ContainerStarted","Data":"1c4be036d4d88306928d8337e10667cf30064b631bc2f41cee2d3adb001cec2e"} Oct 29 23:27:11.923482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:11.915402 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podStartSLOduration=1.9153799 podStartE2EDuration="1.9153799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:11.914779918 +0000 UTC m=+801.277784647" watchObservedRunningTime="2026-10-29 23:27:11.9153799 +0000 UTC m=+801.278384624" Oct 29 23:27:12.093482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:12.090864 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-8gxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 29 23:27:12.093482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:12.090932 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" podUID="b9e3a005-5ced-4440-814c-f2713230bd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 29 23:27:12.139187 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:12.139110 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j" Oct 29 23:27:12.185547 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:12.185514 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-maas-api-openshift-ca\") pod \"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469\" (UID: \"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469\") " Oct 29 23:27:12.185712 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:12.185583 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-kube-api-access-987lh\" (UniqueName: \"kubernetes.io/projected/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-kube-api-access-987lh\") pod \"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469\" (UID: \"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469\") " Oct 29 23:27:12.185712 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:12.185693 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-maas-api-xks-ca\") pod \"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469\" (UID: \"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469\") " Oct 29 23:27:12.185976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:12.185942 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-maas-api-openshift-ca" pod "cde2ebe4-9d4a-4b3f-b684-ed6dc850a469" (UID: "cde2ebe4-9d4a-4b3f-b684-ed6dc850a469"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:12.186205 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:12.186187 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:27:12.187951 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:12.187917 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-maas-api-xks-ca" pod "cde2ebe4-9d4a-4b3f-b684-ed6dc850a469" (UID: "cde2ebe4-9d4a-4b3f-b684-ed6dc850a469"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:27:12.188080 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:12.188002 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-kube-api-access-987lh" pod "cde2ebe4-9d4a-4b3f-b684-ed6dc850a469" (UID: "cde2ebe4-9d4a-4b3f-b684-ed6dc850a469"). InnerVolumeSpecName "kube-api-access-987lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:12.287629 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:12.287596 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-987lh\" (UniqueName: \"kubernetes.io/projected/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-kube-api-access-987lh\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:27:12.287629 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:12.287627 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:27:12.754372 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:12.754342 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:12.894541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:12.889057 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j" Oct 29 23:27:12.894541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:12.893359 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j" event={"ID":"cde2ebe4-9d4a-4b3f-b684-ed6dc850a469","Type":"ContainerDied","Data":"d3996b5a33c98bc5e95b7076eb7842d3f4a00a113d4d51313664a630cc8a6068"} Oct 29 23:27:12.894541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:12.893421 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3996b5a33c98bc5e95b7076eb7842d3f4a00a113d4d51313664a630cc8a6068" Oct 29 23:27:13.089774 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:13.089679 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-8gxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 29 23:27:13.089774 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:13.089748 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" podUID="b9e3a005-5ced-4440-814c-f2713230bd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 29 23:27:14.088965 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:14.088912 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-8gxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 29 23:27:14.089378 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:14.088966 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" podUID="b9e3a005-5ced-4440-814c-f2713230bd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 29 23:27:15.089797 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:15.089760 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-8gxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 29 23:27:15.090219 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:15.089815 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" podUID="b9e3a005-5ced-4440-814c-f2713230bd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 29 23:27:15.249256 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:27:15.249213 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:27:16.088946 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:16.088910 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-8gxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 29 23:27:16.089135 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:16.088964 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" podUID="b9e3a005-5ced-4440-814c-f2713230bd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 29 23:27:17.089261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:17.089224 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-8gxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 29 23:27:17.089783 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:17.089290 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" podUID="b9e3a005-5ced-4440-814c-f2713230bd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 29 23:27:18.088937 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:18.088904 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-8gxl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 29 23:27:18.089123 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:18.088958 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" podUID="b9e3a005-5ced-4440-814c-f2713230bd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 29 23:27:19.296582 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.296525 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m"] Oct 29 23:27:19.297735 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.297707 6507 cpu_manager.go:414] "containerMap: removing container" podUID="cde2ebe4-9d4a-4b3f-b684-ed6dc850a469" containerName="revoke-keys" Oct 29 23:27:19.297735 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.297736 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde2ebe4-9d4a-4b3f-b684-ed6dc850a469" containerName="revoke-keys" Oct 29 23:27:19.297930 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.297909 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde2ebe4-9d4a-4b3f-b684-ed6dc850a469" containerName="revoke-keys" Oct 29 23:27:19.331384 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.306574 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m"] Oct 29 23:27:19.331384 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.327317 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.350326 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.342652 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-be8726\"/\"default-dockercfg-k8nz2\"" Oct 29 23:27:19.350326 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.342795 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-be8726\"/\"distinct-w2-be8726-kserve-self-signed-certs\"" Oct 29 23:27:19.359959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.359369 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-be8726\"/\"kube-root-ca.crt\"" Oct 29 23:27:19.359959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.359495 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-be8726\"/\"openshift-service-ca.crt\"" Oct 29 23:27:19.460576 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.460529 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-home\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.460875 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.460841 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-tmp-dir\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.461076 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.461041 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-model-cache\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.461493 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.461450 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da92dda9-c888-40e8-8790-7d9fc8a86ef7-tls-certs\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.461659 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.461640 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9rwh\" (UniqueName: \"kubernetes.io/projected/da92dda9-c888-40e8-8790-7d9fc8a86ef7-kube-api-access-k9rwh\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.461864 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.461846 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-dshm\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.534516 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.532920 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:19.538182 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.536898 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:19.562541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.549988 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-8gxl6" Oct 29 23:27:19.584484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.563903 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-model-cache\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.584797 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.584762 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-model-cache\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.585201 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.585177 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da92dda9-c888-40e8-8790-7d9fc8a86ef7-tls-certs\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.609485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.585340 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9rwh\" (UniqueName: \"kubernetes.io/projected/da92dda9-c888-40e8-8790-7d9fc8a86ef7-kube-api-access-k9rwh\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.609844 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.600280 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da92dda9-c888-40e8-8790-7d9fc8a86ef7-tls-certs\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.610071 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.610047 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-dshm\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.610672 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.610649 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-home\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.629547 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.629511 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-tmp-dir\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.640378 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.637719 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-tmp-dir\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.640677 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.625725 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-home\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.643833 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.643799 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9rwh\" (UniqueName: \"kubernetes.io/projected/da92dda9-c888-40e8-8790-7d9fc8a86ef7-kube-api-access-k9rwh\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.681490 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.678104 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-dshm\") pod \"distinct-w2-be8726-kserve-56879b868c-lvf8m\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:19.720986 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:19.720703 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:20.137007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:20.136954 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m"] Oct 29 23:27:20.139775 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:27:20.139743 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda92dda9_c888_40e8_8790_7d9fc8a86ef7.slice/crio-ca6dd7e8833d69185d18bf1c543381f4fa44109af77a6985c5edbf9b5926bbe1 WatchSource:0}: Error finding container ca6dd7e8833d69185d18bf1c543381f4fa44109af77a6985c5edbf9b5926bbe1: Status 404 returned error can't find the container with id ca6dd7e8833d69185d18bf1c543381f4fa44109af77a6985c5edbf9b5926bbe1 Oct 29 23:27:20.753400 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:20.753366 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:20.765624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:20.765594 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:20.973819 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:20.973776 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" event={"ID":"da92dda9-c888-40e8-8790-7d9fc8a86ef7","Type":"ContainerStarted","Data":"3d2c97d359c86ed2b7c369b97c94546f9d33db0bdc20bf2781591f2ec77f47f9"} Oct 29 23:27:20.974009 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:20.973832 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" event={"ID":"da92dda9-c888-40e8-8790-7d9fc8a86ef7","Type":"ContainerStarted","Data":"ca6dd7e8833d69185d18bf1c543381f4fa44109af77a6985c5edbf9b5926bbe1"} Oct 29 23:27:20.991086 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:20.991059 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:27:21.122813 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:21.122695 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podStartSLOduration=2.122672415 podStartE2EDuration="2.122672415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:21.031413475 +0000 UTC m=+810.394418199" watchObservedRunningTime="2026-10-29 23:27:21.122672415 +0000 UTC m=+810.485677158" Oct 29 23:27:21.722177 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:21.722139 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:25.150740 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.150704 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4"] Oct 29 23:27:25.155130 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.155108 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.185718 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.185668 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4"] Oct 29 23:27:25.243676 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.243640 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-credential-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.243870 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.243683 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-data\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.243870 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.243714 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-workload-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.243870 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.243792 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-envoy\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.243870 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.243847 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-podinfo\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.244110 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.243874 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-workload-certs\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.244110 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.243958 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-token\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.244110 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.244051 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmcxq\" (UniqueName: \"kubernetes.io/projected/d02ada27-df84-4ea9-988d-3e3f0516f70a-kube-api-access-jmcxq\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.244280 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.244113 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d02ada27-df84-4ea9-988d-3e3f0516f70a-istiod-ca-cert\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.344796 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.344704 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-credential-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.344980 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.344816 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-data\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.344980 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.344842 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-workload-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.344980 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.344880 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-envoy\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.344980 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.344908 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-podinfo\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.344980 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.344946 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-workload-certs\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.345333 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.344982 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-token\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.345333 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.345111 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-credential-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.345333 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.345199 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-data\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.345509 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.345401 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-workload-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.345848 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.345822 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmcxq\" (UniqueName: \"kubernetes.io/projected/d02ada27-df84-4ea9-988d-3e3f0516f70a-kube-api-access-jmcxq\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.345965 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.345907 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d02ada27-df84-4ea9-988d-3e3f0516f70a-istiod-ca-cert\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.346060 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.346041 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-workload-certs\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.346647 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.346626 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d02ada27-df84-4ea9-988d-3e3f0516f70a-istiod-ca-cert\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.347765 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.347737 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-podinfo\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.347765 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.347752 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-envoy\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.365330 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.365288 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-token\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.365540 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.365484 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmcxq\" (UniqueName: \"kubernetes.io/projected/d02ada27-df84-4ea9-988d-3e3f0516f70a-kube-api-access-jmcxq\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.509353 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.509314 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:25.828781 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.828702 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4"] Oct 29 23:27:25.831933 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:27:25.831904 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02ada27_df84_4ea9_988d_3e3f0516f70a.slice/crio-f814bdf20d3d2b3bb0309cee3fe95ee3f4f9b2abfc50f31b23fec5e2a405eb45 WatchSource:0}: Error finding container f814bdf20d3d2b3bb0309cee3fe95ee3f4f9b2abfc50f31b23fec5e2a405eb45: Status 404 returned error can't find the container with id f814bdf20d3d2b3bb0309cee3fe95ee3f4f9b2abfc50f31b23fec5e2a405eb45 Oct 29 23:27:25.834276 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.834243 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:27:25.834351 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.834305 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:27:25.834351 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:25.834340 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:27:26.012651 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:26.012257 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" event={"ID":"d02ada27-df84-4ea9-988d-3e3f0516f70a","Type":"ContainerStarted","Data":"c659e5a2006be61e7e90307624e50779924c7292f29de693a8c0fc0adafae3c3"} Oct 29 23:27:26.012651 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:26.012335 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" event={"ID":"d02ada27-df84-4ea9-988d-3e3f0516f70a","Type":"ContainerStarted","Data":"f814bdf20d3d2b3bb0309cee3fe95ee3f4f9b2abfc50f31b23fec5e2a405eb45"} Oct 29 23:27:26.087546 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:26.085690 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" podStartSLOduration=1.085666367 podStartE2EDuration="1.085666367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:26.063942778 +0000 UTC m=+815.426947502" watchObservedRunningTime="2026-10-29 23:27:26.085666367 +0000 UTC m=+815.448671099" Oct 29 23:27:26.510432 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:26.510389 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:26.512105 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:26.512074 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 29 23:27:26.512235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:26.512126 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" podUID="d02ada27-df84-4ea9-988d-3e3f0516f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 29 23:27:27.510864 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:27.510819 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 29 23:27:27.511536 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:27.511494 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" podUID="d02ada27-df84-4ea9-988d-3e3f0516f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 29 23:27:28.516483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:28.513045 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 29 23:27:28.516483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:28.513115 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" podUID="d02ada27-df84-4ea9-988d-3e3f0516f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 29 23:27:28.973588 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:28.973550 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx"] Oct 29 23:27:28.978346 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:28.978320 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.010611 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.010579 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-313d9c\"/\"distinct-2-w1-313d9c-kserve-self-signed-certs\"" Oct 29 23:27:29.036347 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.036299 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx"] Oct 29 23:27:29.096765 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.096734 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-home\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.096913 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.096770 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8372b3e4-533f-4edc-a90c-91b608494d18-tls-certs\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.096913 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.096804 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-model-cache\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.097026 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.096996 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-dshm\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.097064 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.097048 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-tmp-dir\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.097122 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.097104 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfq5p\" (UniqueName: \"kubernetes.io/projected/8372b3e4-533f-4edc-a90c-91b608494d18-kube-api-access-tfq5p\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.198518 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.198480 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-home\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.198696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.198528 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8372b3e4-533f-4edc-a90c-91b608494d18-tls-certs\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.198696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.198568 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-model-cache\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.198696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.198634 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-dshm\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.198696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.198660 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-tmp-dir\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.198913 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.198719 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfq5p\" (UniqueName: \"kubernetes.io/projected/8372b3e4-533f-4edc-a90c-91b608494d18-kube-api-access-tfq5p\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.198913 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.198865 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-home\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.198986 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.198920 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-model-cache\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.199091 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.199065 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-tmp-dir\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.200823 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.200797 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-dshm\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.201146 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.201127 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8372b3e4-533f-4edc-a90c-91b608494d18-tls-certs\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.240522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.233156 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfq5p\" (UniqueName: \"kubernetes.io/projected/8372b3e4-533f-4edc-a90c-91b608494d18-kube-api-access-tfq5p\") pod \"distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.318541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.313726 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:29.511054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.510512 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 29 23:27:29.511054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.510582 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" podUID="d02ada27-df84-4ea9-988d-3e3f0516f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 29 23:27:29.725842 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.725568 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:29.753030 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.750474 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx"] Oct 29 23:27:29.756164 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:29.756141 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:30.059579 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:30.059444 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" event={"ID":"8372b3e4-533f-4edc-a90c-91b608494d18","Type":"ContainerStarted","Data":"5caba384c63c1977c4f8c8158c9c364d4df55f4de2634df1cf5671fb7813cf9d"} Oct 29 23:27:30.059579 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:30.059532 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" event={"ID":"8372b3e4-533f-4edc-a90c-91b608494d18","Type":"ContainerStarted","Data":"bd6a93e572aed66cd2c9ad81b4d2fc0266c4900dbbd05578ab89a9f03a214743"} Oct 29 23:27:30.071639 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:30.071606 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:27:30.094901 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:30.094850 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podStartSLOduration=2.094835172 podStartE2EDuration="2.094835172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:30.093510193 +0000 UTC m=+819.456514912" watchObservedRunningTime="2026-10-29 23:27:30.094835172 +0000 UTC m=+819.457839895" Oct 29 23:27:30.333556 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:30.320556 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:30.524549 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:30.520987 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 29 23:27:30.524549 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:30.521055 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" podUID="d02ada27-df84-4ea9-988d-3e3f0516f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 29 23:27:31.517483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:31.512968 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 29 23:27:31.517483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:31.513034 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" podUID="d02ada27-df84-4ea9-988d-3e3f0516f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 29 23:27:31.799186 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:31.799102 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:27:32.509982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:32.509945 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 29 23:27:32.510178 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:32.510000 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" podUID="d02ada27-df84-4ea9-988d-3e3f0516f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 29 23:27:33.812510 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:33.808944 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:33.812510 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:33.810571 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:33.812510 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:33.812091 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:27:37.183182 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.183136 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k"] Oct 29 23:27:37.188222 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.188193 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.211334 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.211299 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-837245\"/\"distinct-w6-837245-kserve-self-signed-certs\"" Oct 29 23:27:37.214797 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.214762 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k"] Oct 29 23:27:37.241894 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.241851 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc9f6\" (UniqueName: \"kubernetes.io/projected/bb0c6b55-eae3-4617-a503-ab72e111c5fa-kube-api-access-tc9f6\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.242122 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.242025 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-home\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.242122 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.242115 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-dshm\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.242254 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.242194 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-model-cache\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.242319 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.242257 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb0c6b55-eae3-4617-a503-ab72e111c5fa-tls-certs\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.242319 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.242301 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-tmp-dir\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.352503 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.350285 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc9f6\" (UniqueName: \"kubernetes.io/projected/bb0c6b55-eae3-4617-a503-ab72e111c5fa-kube-api-access-tc9f6\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.352503 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.350402 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-home\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.352503 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.350476 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-dshm\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.352503 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.350557 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-model-cache\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.352503 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.350614 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb0c6b55-eae3-4617-a503-ab72e111c5fa-tls-certs\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.352503 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.350658 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-tmp-dir\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.352503 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.351286 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-tmp-dir\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.352503 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.351898 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-home\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.376208 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.362756 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-model-cache\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.376208 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.372212 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb0c6b55-eae3-4617-a503-ab72e111c5fa-tls-certs\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.385483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.380419 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-dshm\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.385483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.383173 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc9f6\" (UniqueName: \"kubernetes.io/projected/bb0c6b55-eae3-4617-a503-ab72e111c5fa-kube-api-access-tc9f6\") pod \"distinct-w6-837245-kserve-545b6454bd-js42k\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.502228 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.502193 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:37.809324 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:37.809287 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k"] Oct 29 23:27:37.812892 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:27:37.812864 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb0c6b55_eae3_4617_a503_ab72e111c5fa.slice/crio-f858fc8dbb3ce5ee5f30628a7e2c2ac0c242b5977abc06c8f820b6967e9d9a64 WatchSource:0}: Error finding container f858fc8dbb3ce5ee5f30628a7e2c2ac0c242b5977abc06c8f820b6967e9d9a64: Status 404 returned error can't find the container with id f858fc8dbb3ce5ee5f30628a7e2c2ac0c242b5977abc06c8f820b6967e9d9a64 Oct 29 23:27:38.162001 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:38.161915 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" event={"ID":"bb0c6b55-eae3-4617-a503-ab72e111c5fa","Type":"ContainerStarted","Data":"4817da9f0cc3a031e8d9134c698a5ea46c41e9db506f76343d3f318a85e8d444"} Oct 29 23:27:38.162001 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:38.161958 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" event={"ID":"bb0c6b55-eae3-4617-a503-ab72e111c5fa","Type":"ContainerStarted","Data":"f858fc8dbb3ce5ee5f30628a7e2c2ac0c242b5977abc06c8f820b6967e9d9a64"} Oct 29 23:27:38.502403 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:38.502371 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:39.314342 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:39.314299 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:39.326971 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:39.326939 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:39.362902 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:39.362851 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podStartSLOduration=2.362834625 podStartE2EDuration="2.362834625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:38.208985424 +0000 UTC m=+827.571990160" watchObservedRunningTime="2026-10-29 23:27:39.362834625 +0000 UTC m=+828.725839347" Oct 29 23:27:40.029447 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.029404 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2"] Oct 29 23:27:40.033657 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.033641 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.057620 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.057584 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2"] Oct 29 23:27:40.100745 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.100710 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-workload-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.100921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.100830 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f4ed99-042f-4e41-8489-9af7622e632c-istio-podinfo\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.100921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.100904 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-workload-certs\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.100999 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.100988 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-istio-envoy\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.101062 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.101049 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-istio-data\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.101104 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.101076 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f4ed99-042f-4e41-8489-9af7622e632c-istiod-ca-cert\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.101145 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.101124 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f4ed99-042f-4e41-8489-9af7622e632c-istio-token\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.101180 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.101150 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2qdv\" (UniqueName: \"kubernetes.io/projected/12f4ed99-042f-4e41-8489-9af7622e632c-kube-api-access-h2qdv\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.101218 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.101194 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-credential-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.186239 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.186210 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:27:40.201988 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.201960 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-istio-data\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.202087 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.201992 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f4ed99-042f-4e41-8489-9af7622e632c-istiod-ca-cert\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.202087 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.202064 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f4ed99-042f-4e41-8489-9af7622e632c-istio-token\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.202190 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.202099 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2qdv\" (UniqueName: \"kubernetes.io/projected/12f4ed99-042f-4e41-8489-9af7622e632c-kube-api-access-h2qdv\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.202190 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.202131 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-credential-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.202300 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.202228 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-workload-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.202389 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.202367 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-istio-data\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.202450 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.202376 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f4ed99-042f-4e41-8489-9af7622e632c-istio-podinfo\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.202520 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.202446 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-workload-certs\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.202602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.202581 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-istio-envoy\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.202667 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.202626 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-credential-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.202667 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.202648 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-workload-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.202795 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.202748 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-workload-certs\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.202874 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.202850 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f4ed99-042f-4e41-8489-9af7622e632c-istiod-ca-cert\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.204947 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.204922 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-istio-envoy\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.205071 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.204933 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f4ed99-042f-4e41-8489-9af7622e632c-istio-podinfo\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.241248 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.241215 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2qdv\" (UniqueName: \"kubernetes.io/projected/12f4ed99-042f-4e41-8489-9af7622e632c-kube-api-access-h2qdv\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.242409 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.242388 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f4ed99-042f-4e41-8489-9af7622e632c-istio-token\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.348728 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.348648 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:40.683446 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.683419 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2"] Oct 29 23:27:40.685531 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:27:40.685499 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f4ed99_042f_4e41_8489_9af7622e632c.slice/crio-b4641f1981bddb9c4153fcbdb927234c8e3c85d8b1f42b630f85a415264103ad WatchSource:0}: Error finding container b4641f1981bddb9c4153fcbdb927234c8e3c85d8b1f42b630f85a415264103ad: Status 404 returned error can't find the container with id b4641f1981bddb9c4153fcbdb927234c8e3c85d8b1f42b630f85a415264103ad Oct 29 23:27:40.688063 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.688032 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:27:40.688126 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.688096 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:27:40.688161 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:40.688125 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:27:41.183627 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:41.183577 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" event={"ID":"12f4ed99-042f-4e41-8489-9af7622e632c","Type":"ContainerStarted","Data":"2a6cecff018d66364ad7f5d48d752ab3eb2f23b50f5814c81aa66387bfa2ee53"} Oct 29 23:27:41.184117 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:41.183643 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" event={"ID":"12f4ed99-042f-4e41-8489-9af7622e632c","Type":"ContainerStarted","Data":"b4641f1981bddb9c4153fcbdb927234c8e3c85d8b1f42b630f85a415264103ad"} Oct 29 23:27:41.224479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:41.224244 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" podStartSLOduration=1.224222111 podStartE2EDuration="1.224222111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:41.221967569 +0000 UTC m=+830.584972314" watchObservedRunningTime="2026-10-29 23:27:41.224222111 +0000 UTC m=+830.587226835" Oct 29 23:27:41.349775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:41.349742 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:41.351626 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:41.351600 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 29 23:27:41.351758 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:41.351650 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" podUID="12f4ed99-042f-4e41-8489-9af7622e632c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 29 23:27:42.349835 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:42.349800 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 29 23:27:42.350231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:42.349865 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" podUID="12f4ed99-042f-4e41-8489-9af7622e632c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 29 23:27:43.349705 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:43.349668 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 29 23:27:43.349916 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:43.349729 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" podUID="12f4ed99-042f-4e41-8489-9af7622e632c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 29 23:27:44.350032 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:44.349989 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 29 23:27:44.350556 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:44.350051 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" podUID="12f4ed99-042f-4e41-8489-9af7622e632c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 29 23:27:45.249530 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:27:45.249492 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:27:45.349825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:45.349785 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 29 23:27:45.350012 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:45.349847 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" podUID="12f4ed99-042f-4e41-8489-9af7622e632c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 29 23:27:46.349688 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:46.349648 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 29 23:27:46.350108 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:46.349718 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" podUID="12f4ed99-042f-4e41-8489-9af7622e632c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 29 23:27:47.420104 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:47.420071 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:47.420938 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:47.420915 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:47.422284 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:47.422255 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:27:47.503194 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:47.503161 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:47.520431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:47.520402 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:48.270544 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:48.270514 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:27:51.179562 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.179527 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5"] Oct 29 23:27:51.185485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.185432 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" Oct 29 23:27:51.270347 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.270312 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:27:51.393067 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.271787 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:27:51.393067 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.296691 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:27:51.393067 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.332841 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e700ea3-be62-40d2-b685-99dd65a4289d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ad5379-pkjj5\" (UID: \"4e700ea3-be62-40d2-b685-99dd65a4289d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" Oct 29 23:27:51.393357 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.393198 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn7w9\" (UniqueName: \"kubernetes.io/projected/4e700ea3-be62-40d2-b685-99dd65a4289d-kube-api-access-rn7w9\") pod \"maas-api-revoke-sub-c1ad5379-pkjj5\" (UID: \"4e700ea3-be62-40d2-b685-99dd65a4289d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" Oct 29 23:27:51.393357 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.393273 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e700ea3-be62-40d2-b685-99dd65a4289d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ad5379-pkjj5\" (UID: \"4e700ea3-be62-40d2-b685-99dd65a4289d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" Oct 29 23:27:51.393612 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.392822 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:27:51.400561 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.400533 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5"] Oct 29 23:27:51.515509 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.506286 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn7w9\" (UniqueName: \"kubernetes.io/projected/4e700ea3-be62-40d2-b685-99dd65a4289d-kube-api-access-rn7w9\") pod \"maas-api-revoke-sub-c1ad5379-pkjj5\" (UID: \"4e700ea3-be62-40d2-b685-99dd65a4289d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" Oct 29 23:27:51.515509 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.506367 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e700ea3-be62-40d2-b685-99dd65a4289d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ad5379-pkjj5\" (UID: \"4e700ea3-be62-40d2-b685-99dd65a4289d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" Oct 29 23:27:51.515509 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.506601 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e700ea3-be62-40d2-b685-99dd65a4289d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ad5379-pkjj5\" (UID: \"4e700ea3-be62-40d2-b685-99dd65a4289d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" Oct 29 23:27:51.515509 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.508108 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e700ea3-be62-40d2-b685-99dd65a4289d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ad5379-pkjj5\" (UID: \"4e700ea3-be62-40d2-b685-99dd65a4289d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" Oct 29 23:27:51.517741 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.517710 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e700ea3-be62-40d2-b685-99dd65a4289d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ad5379-pkjj5\" (UID: \"4e700ea3-be62-40d2-b685-99dd65a4289d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" Oct 29 23:27:51.524312 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.524288 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn7w9\" (UniqueName: \"kubernetes.io/projected/4e700ea3-be62-40d2-b685-99dd65a4289d-kube-api-access-rn7w9\") pod \"maas-api-revoke-sub-c1ad5379-pkjj5\" (UID: \"4e700ea3-be62-40d2-b685-99dd65a4289d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" Oct 29 23:27:51.819248 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:51.818005 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" Oct 29 23:27:52.445980 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:52.445558 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5"] Oct 29 23:27:52.542660 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:52.542623 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" event={"ID":"4e700ea3-be62-40d2-b685-99dd65a4289d","Type":"ContainerStarted","Data":"b056b6b319eefb3684fafbcbc01ff016e6338387502463270a344a3d9d7501f7"} Oct 29 23:27:53.556138 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:53.556106 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" event={"ID":"4e700ea3-be62-40d2-b685-99dd65a4289d","Type":"ContainerStarted","Data":"794be87a9b78f990e7fc40197ef2874591420656b3902017811406e6b5fdb378"} Oct 29 23:27:53.594388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:53.594317 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" podStartSLOduration=2.594295217 podStartE2EDuration="2.594295217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:53.591151343 +0000 UTC m=+842.954156067" watchObservedRunningTime="2026-10-29 23:27:53.594295217 +0000 UTC m=+842.957299941" Oct 29 23:27:54.568533 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:54.568492 6507 generic.go:356] "Generic (PLEG): container finished" podID="4e700ea3-be62-40d2-b685-99dd65a4289d" containerID="794be87a9b78f990e7fc40197ef2874591420656b3902017811406e6b5fdb378" exitCode=0 Oct 29 23:27:54.568976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:54.568672 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" event={"ID":"4e700ea3-be62-40d2-b685-99dd65a4289d","Type":"ContainerDied","Data":"794be87a9b78f990e7fc40197ef2874591420656b3902017811406e6b5fdb378"} Oct 29 23:27:55.723468 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:55.723431 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" Oct 29 23:27:55.830423 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:55.830333 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e700ea3-be62-40d2-b685-99dd65a4289d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e700ea3-be62-40d2-b685-99dd65a4289d-maas-api-xks-ca\") pod \"4e700ea3-be62-40d2-b685-99dd65a4289d\" (UID: \"4e700ea3-be62-40d2-b685-99dd65a4289d\") " Oct 29 23:27:55.830423 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:55.830379 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e700ea3-be62-40d2-b685-99dd65a4289d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e700ea3-be62-40d2-b685-99dd65a4289d-maas-api-openshift-ca\") pod \"4e700ea3-be62-40d2-b685-99dd65a4289d\" (UID: \"4e700ea3-be62-40d2-b685-99dd65a4289d\") " Oct 29 23:27:55.830634 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:55.830546 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e700ea3-be62-40d2-b685-99dd65a4289d-kube-api-access-rn7w9\" (UniqueName: \"kubernetes.io/projected/4e700ea3-be62-40d2-b685-99dd65a4289d-kube-api-access-rn7w9\") pod \"4e700ea3-be62-40d2-b685-99dd65a4289d\" (UID: \"4e700ea3-be62-40d2-b685-99dd65a4289d\") " Oct 29 23:27:55.830837 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:55.830806 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e700ea3-be62-40d2-b685-99dd65a4289d-maas-api-openshift-ca" pod "4e700ea3-be62-40d2-b685-99dd65a4289d" (UID: "4e700ea3-be62-40d2-b685-99dd65a4289d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:55.831098 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:55.831080 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e700ea3-be62-40d2-b685-99dd65a4289d-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:27:55.832599 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:55.832572 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e700ea3-be62-40d2-b685-99dd65a4289d-maas-api-xks-ca" pod "4e700ea3-be62-40d2-b685-99dd65a4289d" (UID: "4e700ea3-be62-40d2-b685-99dd65a4289d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:27:55.832599 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:55.832588 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e700ea3-be62-40d2-b685-99dd65a4289d-kube-api-access-rn7w9" pod "4e700ea3-be62-40d2-b685-99dd65a4289d" (UID: "4e700ea3-be62-40d2-b685-99dd65a4289d"). InnerVolumeSpecName "kube-api-access-rn7w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:55.932412 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:55.932358 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e700ea3-be62-40d2-b685-99dd65a4289d-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:27:55.932412 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:55.932410 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn7w9\" (UniqueName: \"kubernetes.io/projected/4e700ea3-be62-40d2-b685-99dd65a4289d-kube-api-access-rn7w9\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:27:56.583396 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:56.583367 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" Oct 29 23:27:56.583599 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:56.583362 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5" event={"ID":"4e700ea3-be62-40d2-b685-99dd65a4289d","Type":"ContainerDied","Data":"b056b6b319eefb3684fafbcbc01ff016e6338387502463270a344a3d9d7501f7"} Oct 29 23:27:56.583599 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:27:56.583500 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b056b6b319eefb3684fafbcbc01ff016e6338387502463270a344a3d9d7501f7" Oct 29 23:28:02.711153 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.711116 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz"] Oct 29 23:28:02.711987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.711966 6507 cpu_manager.go:414] "containerMap: removing container" podUID="4e700ea3-be62-40d2-b685-99dd65a4289d" containerName="revoke-keys" Oct 29 23:28:02.711987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.711987 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e700ea3-be62-40d2-b685-99dd65a4289d" containerName="revoke-keys" Oct 29 23:28:02.712133 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.712100 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e700ea3-be62-40d2-b685-99dd65a4289d" containerName="revoke-keys" Oct 29 23:28:02.716747 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.716727 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.725180 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.725150 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53490ca-3cd8-425e-adfd-cf31eec83280-tls-certs\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.725324 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.725295 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-home\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.725401 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.725333 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z8g2\" (UniqueName: \"kubernetes.io/projected/f53490ca-3cd8-425e-adfd-cf31eec83280-kube-api-access-4z8g2\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.725494 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.725406 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-model-cache\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.725565 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.725503 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-tmp-dir\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.725565 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.725533 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-dshm\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.735112 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.735078 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz"] Oct 29 23:28:02.740601 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.740560 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-be8726\"/\"distinct-2-w2-be8726-kserve-self-signed-certs\"" Oct 29 23:28:02.826205 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.826162 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-model-cache\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.826418 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.826228 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-tmp-dir\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.826418 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.826255 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-dshm\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.826418 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.826309 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53490ca-3cd8-425e-adfd-cf31eec83280-tls-certs\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.826678 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.826493 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-home\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.826678 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.826522 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z8g2\" (UniqueName: \"kubernetes.io/projected/f53490ca-3cd8-425e-adfd-cf31eec83280-kube-api-access-4z8g2\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.826766 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.826680 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-model-cache\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.827039 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.826956 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-tmp-dir\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.827039 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.826995 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-home\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.828745 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.828723 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-dshm\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.829004 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.828984 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53490ca-3cd8-425e-adfd-cf31eec83280-tls-certs\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:02.843248 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:02.843220 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z8g2\" (UniqueName: \"kubernetes.io/projected/f53490ca-3cd8-425e-adfd-cf31eec83280-kube-api-access-4z8g2\") pod \"distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:03.033993 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:03.033837 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:03.077554 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:03.077519 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:28:03.348415 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:03.348386 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz"] Oct 29 23:28:03.350516 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:28:03.350488 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf53490ca_3cd8_425e_adfd_cf31eec83280.slice/crio-139d119024c790204866e951586d6c08dfa361eef423ad5e14821ac9a71feceb WatchSource:0}: Error finding container 139d119024c790204866e951586d6c08dfa361eef423ad5e14821ac9a71feceb: Status 404 returned error can't find the container with id 139d119024c790204866e951586d6c08dfa361eef423ad5e14821ac9a71feceb Oct 29 23:28:03.630477 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:03.630369 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" event={"ID":"f53490ca-3cd8-425e-adfd-cf31eec83280","Type":"ContainerStarted","Data":"ea7e85606c419a74ba75728d986f8818442d08c11c3712ca725a9c2ba2d410bc"} Oct 29 23:28:03.630477 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:03.630416 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" event={"ID":"f53490ca-3cd8-425e-adfd-cf31eec83280","Type":"ContainerStarted","Data":"139d119024c790204866e951586d6c08dfa361eef423ad5e14821ac9a71feceb"} Oct 29 23:28:03.669935 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:03.669880 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podStartSLOduration=1.669866933 podStartE2EDuration="1.669866933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:03.668994415 +0000 UTC m=+853.031999137" watchObservedRunningTime="2026-10-29 23:28:03.669866933 +0000 UTC m=+853.032871655" Oct 29 23:28:04.038479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:04.037839 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:06.304788 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.304741 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk"] Oct 29 23:28:06.309197 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.309168 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.330763 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.330729 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-032fa8\"/\"distinct-2-w4-032fa8-kserve-self-signed-certs\"" Oct 29 23:28:06.336395 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.336359 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk"] Oct 29 23:28:06.394407 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.394370 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46fb6a01-f848-4ae7-86dc-26e45162b41c-tls-certs\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.394407 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.394409 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-model-cache\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.394695 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.394611 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncsjr\" (UniqueName: \"kubernetes.io/projected/46fb6a01-f848-4ae7-86dc-26e45162b41c-kube-api-access-ncsjr\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.394695 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.394673 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-home\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.394813 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.394757 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-tmp-dir\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.394891 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.394869 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-dshm\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.531724 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.531682 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-home\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.532012 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.531993 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-tmp-dir\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.532610 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.532585 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-dshm\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.533243 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.533195 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46fb6a01-f848-4ae7-86dc-26e45162b41c-tls-certs\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.538051 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.538029 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-model-cache\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.543171 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.538448 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncsjr\" (UniqueName: \"kubernetes.io/projected/46fb6a01-f848-4ae7-86dc-26e45162b41c-kube-api-access-ncsjr\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.543537 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.542829 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-model-cache\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.547554 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.532194 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-home\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.547719 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.537972 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-tmp-dir\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.561198 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.561123 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46fb6a01-f848-4ae7-86dc-26e45162b41c-tls-certs\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.569802 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.569765 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-dshm\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.572996 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.572960 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncsjr\" (UniqueName: \"kubernetes.io/projected/46fb6a01-f848-4ae7-86dc-26e45162b41c-kube-api-access-ncsjr\") pod \"distinct-2-w4-032fa8-kserve-85b4c49474-25lmk\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.629968 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.629913 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:06.936939 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:06.936907 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk"] Oct 29 23:28:06.937989 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:28:06.937954 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46fb6a01_f848_4ae7_86dc_26e45162b41c.slice/crio-ce36bdd9f3dcf051d951bd7ddd0eae8dde7495c33e720925f8b6c9ab35b7df85 WatchSource:0}: Error finding container ce36bdd9f3dcf051d951bd7ddd0eae8dde7495c33e720925f8b6c9ab35b7df85: Status 404 returned error can't find the container with id ce36bdd9f3dcf051d951bd7ddd0eae8dde7495c33e720925f8b6c9ab35b7df85 Oct 29 23:28:07.663968 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:07.663929 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" event={"ID":"46fb6a01-f848-4ae7-86dc-26e45162b41c","Type":"ContainerStarted","Data":"e51b6ce1ed688be641124d0f999c15714d4b1c652424539905596a93cd72dbd1"} Oct 29 23:28:07.663968 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:07.663976 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" event={"ID":"46fb6a01-f848-4ae7-86dc-26e45162b41c","Type":"ContainerStarted","Data":"ce36bdd9f3dcf051d951bd7ddd0eae8dde7495c33e720925f8b6c9ab35b7df85"} Oct 29 23:28:07.701911 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:07.701847 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podStartSLOduration=1.701826857 podStartE2EDuration="1.701826857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:07.698916839 +0000 UTC m=+857.061921559" watchObservedRunningTime="2026-10-29 23:28:07.701826857 +0000 UTC m=+857.064831579" Oct 29 23:28:08.630392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:08.630351 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:09.280322 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.280288 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd"] Oct 29 23:28:09.287300 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.287275 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.314356 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.314320 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd"] Oct 29 23:28:09.383027 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.382993 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-envoy\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.383198 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.383032 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j687m\" (UniqueName: \"kubernetes.io/projected/991d97b5-f7ae-4895-a247-79b1200eec8b-kube-api-access-j687m\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.383295 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.383209 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-podinfo\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.383295 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.383255 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-workload-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.383371 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.383320 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-credential-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.383420 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.383404 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-data\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.383469 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.383447 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-token\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.383539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.383526 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-workload-certs\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.383592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.383573 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991d97b5-f7ae-4895-a247-79b1200eec8b-istiod-ca-cert\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.484981 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.484941 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-workload-certs\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.485183 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.484992 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991d97b5-f7ae-4895-a247-79b1200eec8b-istiod-ca-cert\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.485183 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.485068 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-envoy\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.485183 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.485086 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j687m\" (UniqueName: \"kubernetes.io/projected/991d97b5-f7ae-4895-a247-79b1200eec8b-kube-api-access-j687m\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.485183 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.485141 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-podinfo\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.485183 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.485164 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-workload-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.485639 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.485197 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-credential-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.485639 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.485268 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-data\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.485639 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.485326 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-token\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.485787 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.485714 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-workload-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.485951 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.485924 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-data\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.486028 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.485973 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-credential-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.486207 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.486181 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-workload-certs\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.486340 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.486316 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991d97b5-f7ae-4895-a247-79b1200eec8b-istiod-ca-cert\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.488277 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.488254 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-envoy\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.488820 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.488800 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-podinfo\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.501837 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.501806 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j687m\" (UniqueName: \"kubernetes.io/projected/991d97b5-f7ae-4895-a247-79b1200eec8b-kube-api-access-j687m\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.502262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.502242 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-token\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:09.602395 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:09.602315 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:10.013342 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:10.013312 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd"] Oct 29 23:28:10.016487 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:28:10.016443 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991d97b5_f7ae_4895_a247_79b1200eec8b.slice/crio-f65e7ba29d77b27092f530e71c8839e2f29d65ef1007c016dccf7d7736045744 WatchSource:0}: Error finding container f65e7ba29d77b27092f530e71c8839e2f29d65ef1007c016dccf7d7736045744: Status 404 returned error can't find the container with id f65e7ba29d77b27092f530e71c8839e2f29d65ef1007c016dccf7d7736045744 Oct 29 23:28:10.019251 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:10.019216 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:28:10.019326 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:10.019289 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:28:10.019326 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:10.019317 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:28:10.768250 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:10.768217 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" event={"ID":"991d97b5-f7ae-4895-a247-79b1200eec8b","Type":"ContainerStarted","Data":"2371d91b0f42e9f41f5950256024ad0a2eda950abe01831c2315dd48bdbf923f"} Oct 29 23:28:10.768690 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:10.768271 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" event={"ID":"991d97b5-f7ae-4895-a247-79b1200eec8b","Type":"ContainerStarted","Data":"f65e7ba29d77b27092f530e71c8839e2f29d65ef1007c016dccf7d7736045744"} Oct 29 23:28:10.835999 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:10.835946 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" podStartSLOduration=1.835932943 podStartE2EDuration="1.835932943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:10.832384253 +0000 UTC m=+860.195388976" watchObservedRunningTime="2026-10-29 23:28:10.835932943 +0000 UTC m=+860.198937665" Oct 29 23:28:11.608380 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:11.605842 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:11.626539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:11.613588 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 23:28:11.626539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:11.613650 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" podUID="991d97b5-f7ae-4895-a247-79b1200eec8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 23:28:12.608482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:12.605853 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 23:28:12.608482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:12.605915 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" podUID="991d97b5-f7ae-4895-a247-79b1200eec8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 23:28:13.035020 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:13.034987 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:13.047822 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:13.047792 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:13.624485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:13.621593 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 23:28:13.624485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:13.621656 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" podUID="991d97b5-f7ae-4895-a247-79b1200eec8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 23:28:13.827282 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:13.827247 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:14.620482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:14.611847 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 23:28:14.620482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:14.611922 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" podUID="991d97b5-f7ae-4895-a247-79b1200eec8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 23:28:15.248150 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:28:15.248114 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:28:15.605532 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:15.604960 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 23:28:15.605532 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:15.605022 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" podUID="991d97b5-f7ae-4895-a247-79b1200eec8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 23:28:16.603792 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:16.603755 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 23:28:16.604198 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:16.603814 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" podUID="991d97b5-f7ae-4895-a247-79b1200eec8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 23:28:16.631177 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:16.631143 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:16.644206 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:16.644174 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:16.847502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:16.847439 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:17.607009 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:17.606977 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:17.607509 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:17.607490 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:17.608302 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:17.608281 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:28:20.435033 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.434995 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx"] Oct 29 23:28:20.439834 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.439814 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" Oct 29 23:28:20.465773 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.465744 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx"] Oct 29 23:28:20.470731 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.470708 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:28:20.470875 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.470798 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:28:20.470875 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.470865 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:28:20.470993 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.470933 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:28:20.528581 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.528533 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4410de-1052-48de-8eb5-9380c75b35a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a18c96f4-q8nhx\" (UID: \"ec4410de-1052-48de-8eb5-9380c75b35a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" Oct 29 23:28:20.528762 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.528677 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4410de-1052-48de-8eb5-9380c75b35a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a18c96f4-q8nhx\" (UID: \"ec4410de-1052-48de-8eb5-9380c75b35a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" Oct 29 23:28:20.528762 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.528702 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7kcx\" (UniqueName: \"kubernetes.io/projected/ec4410de-1052-48de-8eb5-9380c75b35a8-kube-api-access-w7kcx\") pod \"maas-api-revoke-sub-a18c96f4-q8nhx\" (UID: \"ec4410de-1052-48de-8eb5-9380c75b35a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" Oct 29 23:28:20.630381 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.630347 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4410de-1052-48de-8eb5-9380c75b35a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a18c96f4-q8nhx\" (UID: \"ec4410de-1052-48de-8eb5-9380c75b35a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" Oct 29 23:28:20.630590 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.630438 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4410de-1052-48de-8eb5-9380c75b35a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a18c96f4-q8nhx\" (UID: \"ec4410de-1052-48de-8eb5-9380c75b35a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" Oct 29 23:28:20.630590 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.630476 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7kcx\" (UniqueName: \"kubernetes.io/projected/ec4410de-1052-48de-8eb5-9380c75b35a8-kube-api-access-w7kcx\") pod \"maas-api-revoke-sub-a18c96f4-q8nhx\" (UID: \"ec4410de-1052-48de-8eb5-9380c75b35a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" Oct 29 23:28:20.631209 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.631187 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4410de-1052-48de-8eb5-9380c75b35a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a18c96f4-q8nhx\" (UID: \"ec4410de-1052-48de-8eb5-9380c75b35a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" Oct 29 23:28:20.633050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.633027 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4410de-1052-48de-8eb5-9380c75b35a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a18c96f4-q8nhx\" (UID: \"ec4410de-1052-48de-8eb5-9380c75b35a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" Oct 29 23:28:20.647512 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.647481 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7kcx\" (UniqueName: \"kubernetes.io/projected/ec4410de-1052-48de-8eb5-9380c75b35a8-kube-api-access-w7kcx\") pod \"maas-api-revoke-sub-a18c96f4-q8nhx\" (UID: \"ec4410de-1052-48de-8eb5-9380c75b35a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" Oct 29 23:28:20.754011 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:20.753923 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" Oct 29 23:28:21.059278 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:21.059249 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx"] Oct 29 23:28:21.061058 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:28:21.061028 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4410de_1052_48de_8eb5_9380c75b35a8.slice/crio-fc55f6440280e24d0da3007396de1c26f7b510a3849f41f39dc9a7600a535f7c WatchSource:0}: Error finding container fc55f6440280e24d0da3007396de1c26f7b510a3849f41f39dc9a7600a535f7c: Status 404 returned error can't find the container with id fc55f6440280e24d0da3007396de1c26f7b510a3849f41f39dc9a7600a535f7c Oct 29 23:28:21.908078 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:21.908035 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" event={"ID":"ec4410de-1052-48de-8eb5-9380c75b35a8","Type":"ContainerStarted","Data":"13b102c610d2c8922fa00579135284f30ba19652e818b6b8b46604f42bebbe6d"} Oct 29 23:28:21.908983 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:21.908916 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" event={"ID":"ec4410de-1052-48de-8eb5-9380c75b35a8","Type":"ContainerStarted","Data":"fc55f6440280e24d0da3007396de1c26f7b510a3849f41f39dc9a7600a535f7c"} Oct 29 23:28:21.983201 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:21.983134 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" podStartSLOduration=1.9831139260000001 podStartE2EDuration="1.983113926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:21.965701426 +0000 UTC m=+871.328706151" watchObservedRunningTime="2026-10-29 23:28:21.983113926 +0000 UTC m=+871.346118652" Oct 29 23:28:22.946139 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:22.946082 6507 generic.go:356] "Generic (PLEG): container finished" podID="ec4410de-1052-48de-8eb5-9380c75b35a8" containerID="13b102c610d2c8922fa00579135284f30ba19652e818b6b8b46604f42bebbe6d" exitCode=0 Oct 29 23:28:22.951019 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:22.946909 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" event={"ID":"ec4410de-1052-48de-8eb5-9380c75b35a8","Type":"ContainerDied","Data":"13b102c610d2c8922fa00579135284f30ba19652e818b6b8b46604f42bebbe6d"} Oct 29 23:28:24.343817 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.343786 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" Oct 29 23:28:24.447559 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.447515 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec4410de-1052-48de-8eb5-9380c75b35a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4410de-1052-48de-8eb5-9380c75b35a8-maas-api-xks-ca\") pod \"ec4410de-1052-48de-8eb5-9380c75b35a8\" (UID: \"ec4410de-1052-48de-8eb5-9380c75b35a8\") " Oct 29 23:28:24.447755 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.447626 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec4410de-1052-48de-8eb5-9380c75b35a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4410de-1052-48de-8eb5-9380c75b35a8-maas-api-openshift-ca\") pod \"ec4410de-1052-48de-8eb5-9380c75b35a8\" (UID: \"ec4410de-1052-48de-8eb5-9380c75b35a8\") " Oct 29 23:28:24.447822 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.447753 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4410de-1052-48de-8eb5-9380c75b35a8-kube-api-access-w7kcx\" (UniqueName: \"kubernetes.io/projected/ec4410de-1052-48de-8eb5-9380c75b35a8-kube-api-access-w7kcx\") pod \"ec4410de-1052-48de-8eb5-9380c75b35a8\" (UID: \"ec4410de-1052-48de-8eb5-9380c75b35a8\") " Oct 29 23:28:24.448044 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.448011 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4410de-1052-48de-8eb5-9380c75b35a8-maas-api-openshift-ca" pod "ec4410de-1052-48de-8eb5-9380c75b35a8" (UID: "ec4410de-1052-48de-8eb5-9380c75b35a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:24.448296 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.448277 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4410de-1052-48de-8eb5-9380c75b35a8-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:28:24.449715 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.449692 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4410de-1052-48de-8eb5-9380c75b35a8-kube-api-access-w7kcx" pod "ec4410de-1052-48de-8eb5-9380c75b35a8" (UID: "ec4410de-1052-48de-8eb5-9380c75b35a8"). InnerVolumeSpecName "kube-api-access-w7kcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:24.449820 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.449693 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec4410de-1052-48de-8eb5-9380c75b35a8-maas-api-xks-ca" pod "ec4410de-1052-48de-8eb5-9380c75b35a8" (UID: "ec4410de-1052-48de-8eb5-9380c75b35a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:24.549833 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.549787 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4410de-1052-48de-8eb5-9380c75b35a8-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:28:24.549833 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.549825 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7kcx\" (UniqueName: \"kubernetes.io/projected/ec4410de-1052-48de-8eb5-9380c75b35a8-kube-api-access-w7kcx\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:28:24.820001 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.819967 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf"] Oct 29 23:28:24.820583 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.820570 6507 cpu_manager.go:414] "containerMap: removing container" podUID="ec4410de-1052-48de-8eb5-9380c75b35a8" containerName="revoke-keys" Oct 29 23:28:24.820632 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.820585 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4410de-1052-48de-8eb5-9380c75b35a8" containerName="revoke-keys" Oct 29 23:28:24.820718 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.820709 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4410de-1052-48de-8eb5-9380c75b35a8" containerName="revoke-keys" Oct 29 23:28:24.824253 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.824238 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:24.869205 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.869173 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf"] Oct 29 23:28:24.967686 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.967649 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7m7m\" (UniqueName: \"kubernetes.io/projected/25fce51a-8d4e-4ac0-87e2-68457f9f10df-kube-api-access-g7m7m\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:24.967686 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.967688 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-workload-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:24.967904 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.967713 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-credential-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:24.967904 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.967826 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-data\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:24.967976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.967958 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-workload-certs\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:24.968013 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.967988 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-envoy\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:24.968053 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.968013 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-token\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:24.968105 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.968082 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-podinfo\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:24.968208 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.968195 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istiod-ca-cert\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:24.990111 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.990082 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" Oct 29 23:28:24.990299 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.990115 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx" event={"ID":"ec4410de-1052-48de-8eb5-9380c75b35a8","Type":"ContainerDied","Data":"fc55f6440280e24d0da3007396de1c26f7b510a3849f41f39dc9a7600a535f7c"} Oct 29 23:28:24.990299 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:24.990167 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc55f6440280e24d0da3007396de1c26f7b510a3849f41f39dc9a7600a535f7c" Oct 29 23:28:25.069710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.069614 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-workload-certs\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.069710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.069663 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-envoy\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.069710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.069700 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-token\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.069990 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.069734 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-podinfo\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.069990 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.069817 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istiod-ca-cert\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.069990 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.069901 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7m7m\" (UniqueName: \"kubernetes.io/projected/25fce51a-8d4e-4ac0-87e2-68457f9f10df-kube-api-access-g7m7m\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.069990 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.069933 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-workload-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.069990 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.069966 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-credential-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.070230 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.070006 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-data\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.070230 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.070029 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-workload-certs\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.070396 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.070374 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-data\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.070588 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.070565 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-credential-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.070719 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.070701 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-workload-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.070824 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.070802 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istiod-ca-cert\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.072314 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.072288 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-envoy\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.072785 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.072762 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-podinfo\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.086922 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.086896 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-token\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.087104 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.087083 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7m7m\" (UniqueName: \"kubernetes.io/projected/25fce51a-8d4e-4ac0-87e2-68457f9f10df-kube-api-access-g7m7m\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.157129 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.157089 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:25.490040 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.490012 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf"] Oct 29 23:28:25.492882 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:28:25.492837 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25fce51a_8d4e_4ac0_87e2_68457f9f10df.slice/crio-a10ab5736916bf0df0e4094191275503f4c5d3f8d23579dc97f528aa0aa95213 WatchSource:0}: Error finding container a10ab5736916bf0df0e4094191275503f4c5d3f8d23579dc97f528aa0aa95213: Status 404 returned error can't find the container with id a10ab5736916bf0df0e4094191275503f4c5d3f8d23579dc97f528aa0aa95213 Oct 29 23:28:25.495590 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.495553 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:28:25.495663 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.495633 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:28:25.495709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:25.495664 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:28:26.029908 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:26.029868 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" event={"ID":"25fce51a-8d4e-4ac0-87e2-68457f9f10df","Type":"ContainerStarted","Data":"b8b669694b4318f57f1119742bfbeea79fc5e59b6a08e282100fb62a4b320c56"} Oct 29 23:28:26.031300 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:26.031272 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" event={"ID":"25fce51a-8d4e-4ac0-87e2-68457f9f10df","Type":"ContainerStarted","Data":"a10ab5736916bf0df0e4094191275503f4c5d3f8d23579dc97f528aa0aa95213"} Oct 29 23:28:26.096652 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:26.096583 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" podStartSLOduration=2.096559247 podStartE2EDuration="2.096559247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:26.080317201 +0000 UTC m=+875.443321954" watchObservedRunningTime="2026-10-29 23:28:26.096559247 +0000 UTC m=+875.459563971" Oct 29 23:28:26.159884 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:26.159799 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:26.161387 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:26.161355 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 23:28:26.161599 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:26.161566 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" podUID="25fce51a-8d4e-4ac0-87e2-68457f9f10df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 23:28:27.158349 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:27.158313 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 23:28:27.158752 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:27.158368 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" podUID="25fce51a-8d4e-4ac0-87e2-68457f9f10df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 23:28:28.158139 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:28.158094 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 23:28:28.158315 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:28.158159 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" podUID="25fce51a-8d4e-4ac0-87e2-68457f9f10df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 23:28:29.157630 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:29.157589 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 23:28:29.158028 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:29.157648 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" podUID="25fce51a-8d4e-4ac0-87e2-68457f9f10df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 23:28:30.158628 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:30.158592 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 23:28:30.159049 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:30.158655 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" podUID="25fce51a-8d4e-4ac0-87e2-68457f9f10df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 23:28:31.158053 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:31.158013 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 23:28:31.158237 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:31.158086 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" podUID="25fce51a-8d4e-4ac0-87e2-68457f9f10df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 23:28:32.158272 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:32.158236 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 23:28:32.158717 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:32.158301 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" podUID="25fce51a-8d4e-4ac0-87e2-68457f9f10df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 23:28:32.817447 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:32.817408 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq"] Oct 29 23:28:32.822606 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:32.822580 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" Oct 29 23:28:32.838450 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:32.838419 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:28:32.847781 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:32.847754 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:28:32.848051 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:32.848033 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:28:32.848960 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:32.848930 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:28:32.854096 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:32.854054 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq"] Oct 29 23:28:32.969617 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:32.969584 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda96310-6dec-41a2-8429-1751247210ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28be6ca0-47wkq\" (UID: \"cda96310-6dec-41a2-8429-1751247210ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" Oct 29 23:28:32.969808 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:32.969651 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7v84\" (UniqueName: \"kubernetes.io/projected/cda96310-6dec-41a2-8429-1751247210ab-kube-api-access-p7v84\") pod \"maas-api-revoke-sub-28be6ca0-47wkq\" (UID: \"cda96310-6dec-41a2-8429-1751247210ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" Oct 29 23:28:32.969808 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:32.969793 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda96310-6dec-41a2-8429-1751247210ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28be6ca0-47wkq\" (UID: \"cda96310-6dec-41a2-8429-1751247210ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" Oct 29 23:28:33.071978 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:33.071890 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda96310-6dec-41a2-8429-1751247210ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28be6ca0-47wkq\" (UID: \"cda96310-6dec-41a2-8429-1751247210ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" Oct 29 23:28:33.071978 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:33.071952 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7v84\" (UniqueName: \"kubernetes.io/projected/cda96310-6dec-41a2-8429-1751247210ab-kube-api-access-p7v84\") pod \"maas-api-revoke-sub-28be6ca0-47wkq\" (UID: \"cda96310-6dec-41a2-8429-1751247210ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" Oct 29 23:28:33.072165 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:33.072024 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda96310-6dec-41a2-8429-1751247210ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28be6ca0-47wkq\" (UID: \"cda96310-6dec-41a2-8429-1751247210ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" Oct 29 23:28:33.072580 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:33.072551 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda96310-6dec-41a2-8429-1751247210ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28be6ca0-47wkq\" (UID: \"cda96310-6dec-41a2-8429-1751247210ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" Oct 29 23:28:33.074330 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:33.074308 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda96310-6dec-41a2-8429-1751247210ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28be6ca0-47wkq\" (UID: \"cda96310-6dec-41a2-8429-1751247210ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" Oct 29 23:28:33.088485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:33.088447 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7v84\" (UniqueName: \"kubernetes.io/projected/cda96310-6dec-41a2-8429-1751247210ab-kube-api-access-p7v84\") pod \"maas-api-revoke-sub-28be6ca0-47wkq\" (UID: \"cda96310-6dec-41a2-8429-1751247210ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" Oct 29 23:28:33.137290 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:33.137262 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" Oct 29 23:28:33.162739 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:33.162708 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:33.163179 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:33.163126 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:33.163909 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:33.163887 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:28:33.453922 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:33.453895 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq"] Oct 29 23:28:33.455515 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:28:33.455487 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda96310_6dec_41a2_8429_1751247210ab.slice/crio-4679be53e00a0284c6052a3267f22451c00de5a8729c6d30812d2bb647be1345 WatchSource:0}: Error finding container 4679be53e00a0284c6052a3267f22451c00de5a8729c6d30812d2bb647be1345: Status 404 returned error can't find the container with id 4679be53e00a0284c6052a3267f22451c00de5a8729c6d30812d2bb647be1345 Oct 29 23:28:34.184536 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:34.184226 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" event={"ID":"cda96310-6dec-41a2-8429-1751247210ab","Type":"ContainerStarted","Data":"0ad492ed4611013494b4a903fc6ebf68eec9e849b1cac8246eb5ef50bbba2ebd"} Oct 29 23:28:34.185010 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:34.184678 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" event={"ID":"cda96310-6dec-41a2-8429-1751247210ab","Type":"ContainerStarted","Data":"4679be53e00a0284c6052a3267f22451c00de5a8729c6d30812d2bb647be1345"} Oct 29 23:28:34.225509 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:34.220937 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" podStartSLOduration=2.220914826 podStartE2EDuration="2.220914826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:34.217952423 +0000 UTC m=+883.580957193" watchObservedRunningTime="2026-10-29 23:28:34.220914826 +0000 UTC m=+883.583919550" Oct 29 23:28:35.191113 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:35.191077 6507 generic.go:356] "Generic (PLEG): container finished" podID="cda96310-6dec-41a2-8429-1751247210ab" containerID="0ad492ed4611013494b4a903fc6ebf68eec9e849b1cac8246eb5ef50bbba2ebd" exitCode=0 Oct 29 23:28:35.191589 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:35.191147 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" event={"ID":"cda96310-6dec-41a2-8429-1751247210ab","Type":"ContainerDied","Data":"0ad492ed4611013494b4a903fc6ebf68eec9e849b1cac8246eb5ef50bbba2ebd"} Oct 29 23:28:35.395937 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:35.395895 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf"] Oct 29 23:28:35.399951 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:35.399930 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf" Oct 29 23:28:35.420562 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:35.420525 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf"] Oct 29 23:28:35.514609 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:35.514512 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w84qd\" (UniqueName: \"kubernetes.io/projected/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-kube-api-access-w84qd\") pod \"maas-api-revoke-sub-81beb87f-jzrjf\" (UID: \"49ec50ce-e1cb-4201-9a60-147f1bcb1db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf" Oct 29 23:28:35.514772 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:35.514637 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81beb87f-jzrjf\" (UID: \"49ec50ce-e1cb-4201-9a60-147f1bcb1db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf" Oct 29 23:28:35.514772 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:35.514690 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81beb87f-jzrjf\" (UID: \"49ec50ce-e1cb-4201-9a60-147f1bcb1db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf" Oct 29 23:28:35.616710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:35.616674 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w84qd\" (UniqueName: \"kubernetes.io/projected/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-kube-api-access-w84qd\") pod \"maas-api-revoke-sub-81beb87f-jzrjf\" (UID: \"49ec50ce-e1cb-4201-9a60-147f1bcb1db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf" Oct 29 23:28:35.616710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:35.616720 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81beb87f-jzrjf\" (UID: \"49ec50ce-e1cb-4201-9a60-147f1bcb1db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf" Oct 29 23:28:35.616925 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:35.616740 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81beb87f-jzrjf\" (UID: \"49ec50ce-e1cb-4201-9a60-147f1bcb1db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf" Oct 29 23:28:35.617374 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:35.617342 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81beb87f-jzrjf\" (UID: \"49ec50ce-e1cb-4201-9a60-147f1bcb1db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf" Oct 29 23:28:35.619117 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:35.619095 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81beb87f-jzrjf\" (UID: \"49ec50ce-e1cb-4201-9a60-147f1bcb1db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf" Oct 29 23:28:35.633424 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:35.633390 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w84qd\" (UniqueName: \"kubernetes.io/projected/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-kube-api-access-w84qd\") pod \"maas-api-revoke-sub-81beb87f-jzrjf\" (UID: \"49ec50ce-e1cb-4201-9a60-147f1bcb1db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf" Oct 29 23:28:35.713289 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:35.713249 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf" Oct 29 23:28:36.124958 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:36.124926 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf"] Oct 29 23:28:36.127426 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:28:36.127392 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ec50ce_e1cb_4201_9a60_147f1bcb1db8.slice/crio-8e01a3c37c5eaef3d9cb925905769f8a0c92cbe7a8713c036741bb239a98c68d WatchSource:0}: Error finding container 8e01a3c37c5eaef3d9cb925905769f8a0c92cbe7a8713c036741bb239a98c68d: Status 404 returned error can't find the container with id 8e01a3c37c5eaef3d9cb925905769f8a0c92cbe7a8713c036741bb239a98c68d Oct 29 23:28:36.199382 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:36.199343 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf" event={"ID":"49ec50ce-e1cb-4201-9a60-147f1bcb1db8","Type":"ContainerStarted","Data":"8e01a3c37c5eaef3d9cb925905769f8a0c92cbe7a8713c036741bb239a98c68d"} Oct 29 23:28:36.331541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:36.331515 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" Oct 29 23:28:36.429007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:36.428963 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cda96310-6dec-41a2-8429-1751247210ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda96310-6dec-41a2-8429-1751247210ab-maas-api-xks-ca\") pod \"cda96310-6dec-41a2-8429-1751247210ab\" (UID: \"cda96310-6dec-41a2-8429-1751247210ab\") " Oct 29 23:28:36.429207 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:36.429023 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda96310-6dec-41a2-8429-1751247210ab-kube-api-access-p7v84\" (UniqueName: \"kubernetes.io/projected/cda96310-6dec-41a2-8429-1751247210ab-kube-api-access-p7v84\") pod \"cda96310-6dec-41a2-8429-1751247210ab\" (UID: \"cda96310-6dec-41a2-8429-1751247210ab\") " Oct 29 23:28:36.429254 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:36.429242 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cda96310-6dec-41a2-8429-1751247210ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda96310-6dec-41a2-8429-1751247210ab-maas-api-openshift-ca\") pod \"cda96310-6dec-41a2-8429-1751247210ab\" (UID: \"cda96310-6dec-41a2-8429-1751247210ab\") " Oct 29 23:28:36.429641 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:36.429611 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda96310-6dec-41a2-8429-1751247210ab-maas-api-openshift-ca" pod "cda96310-6dec-41a2-8429-1751247210ab" (UID: "cda96310-6dec-41a2-8429-1751247210ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:36.429779 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:36.429718 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda96310-6dec-41a2-8429-1751247210ab-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:28:36.431168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:36.431139 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cda96310-6dec-41a2-8429-1751247210ab-maas-api-xks-ca" pod "cda96310-6dec-41a2-8429-1751247210ab" (UID: "cda96310-6dec-41a2-8429-1751247210ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:36.431281 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:36.431149 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda96310-6dec-41a2-8429-1751247210ab-kube-api-access-p7v84" pod "cda96310-6dec-41a2-8429-1751247210ab" (UID: "cda96310-6dec-41a2-8429-1751247210ab"). InnerVolumeSpecName "kube-api-access-p7v84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:36.531640 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:36.531606 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda96310-6dec-41a2-8429-1751247210ab-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:28:36.531640 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:36.531639 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7v84\" (UniqueName: \"kubernetes.io/projected/cda96310-6dec-41a2-8429-1751247210ab-kube-api-access-p7v84\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:28:37.239481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:37.238976 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" event={"ID":"cda96310-6dec-41a2-8429-1751247210ab","Type":"ContainerDied","Data":"4679be53e00a0284c6052a3267f22451c00de5a8729c6d30812d2bb647be1345"} Oct 29 23:28:37.239481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:37.239054 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4679be53e00a0284c6052a3267f22451c00de5a8729c6d30812d2bb647be1345" Oct 29 23:28:37.239481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:37.239175 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq" Oct 29 23:28:37.245363 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:37.245330 6507 generic.go:356] "Generic (PLEG): container finished" podID="49ec50ce-e1cb-4201-9a60-147f1bcb1db8" containerID="0c17c5edacd5fd378a2fa8b16019dc4779f3990399a537e77b16b1807277199c" exitCode=0 Oct 29 23:28:37.245553 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:37.245369 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf" event={"ID":"49ec50ce-e1cb-4201-9a60-147f1bcb1db8","Type":"ContainerDied","Data":"0c17c5edacd5fd378a2fa8b16019dc4779f3990399a537e77b16b1807277199c"} Oct 29 23:28:38.217890 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:38.217851 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:28:38.414859 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:38.414830 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf" Oct 29 23:28:38.575720 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:38.566911 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-maas-api-xks-ca\") pod \"49ec50ce-e1cb-4201-9a60-147f1bcb1db8\" (UID: \"49ec50ce-e1cb-4201-9a60-147f1bcb1db8\") " Oct 29 23:28:38.575720 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:38.567265 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-kube-api-access-w84qd\" (UniqueName: \"kubernetes.io/projected/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-kube-api-access-w84qd\") pod \"49ec50ce-e1cb-4201-9a60-147f1bcb1db8\" (UID: \"49ec50ce-e1cb-4201-9a60-147f1bcb1db8\") " Oct 29 23:28:38.575720 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:38.567297 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-maas-api-openshift-ca\") pod \"49ec50ce-e1cb-4201-9a60-147f1bcb1db8\" (UID: \"49ec50ce-e1cb-4201-9a60-147f1bcb1db8\") " Oct 29 23:28:38.587171 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:38.583096 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-maas-api-openshift-ca" pod "49ec50ce-e1cb-4201-9a60-147f1bcb1db8" (UID: "49ec50ce-e1cb-4201-9a60-147f1bcb1db8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:38.588532 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:38.588433 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-maas-api-xks-ca" pod "49ec50ce-e1cb-4201-9a60-147f1bcb1db8" (UID: "49ec50ce-e1cb-4201-9a60-147f1bcb1db8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:38.588532 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:38.588487 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-kube-api-access-w84qd" pod "49ec50ce-e1cb-4201-9a60-147f1bcb1db8" (UID: "49ec50ce-e1cb-4201-9a60-147f1bcb1db8"). InnerVolumeSpecName "kube-api-access-w84qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:38.669238 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:38.669200 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w84qd\" (UniqueName: \"kubernetes.io/projected/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-kube-api-access-w84qd\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:28:38.669238 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:38.669232 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:28:38.669238 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:38.669241 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ec50ce-e1cb-4201-9a60-147f1bcb1db8-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:28:39.296481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:39.295529 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf" event={"ID":"49ec50ce-e1cb-4201-9a60-147f1bcb1db8","Type":"ContainerDied","Data":"8e01a3c37c5eaef3d9cb925905769f8a0c92cbe7a8713c036741bb239a98c68d"} Oct 29 23:28:39.296481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:39.295592 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e01a3c37c5eaef3d9cb925905769f8a0c92cbe7a8713c036741bb239a98c68d" Oct 29 23:28:39.296481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:39.295668 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf" Oct 29 23:28:40.768483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:40.768164 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:41.029542 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:41.025620 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:41.069494 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:41.068552 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:41.186483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:41.180618 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 29 23:28:41.271630 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:41.271573 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:41.698630 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:41.698576 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:41.740670 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:41.740607 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 29 23:28:41.769609 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:41.769563 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:41.770583 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:41.770410 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:28:41.771494 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:41.771396 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:41.800884 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:41.800793 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:41.801234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:41.801206 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:41.801490 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:41.801430 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:41.853514 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:41.849380 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 29 23:28:42.149578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.142241 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:42.149578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.144734 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:28:42.149578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.145614 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 29 23:28:42.157537 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.152808 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:28:42.195042 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.194999 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:42.195672 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.195648 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:28:42.411540 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.390727 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 23:28:42.465484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.445447 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:28:42.745550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.710334 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:42.745550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.711054 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:28:42.782484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.755936 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 29 23:28:42.782484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.756328 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:28:42.783274 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.783235 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 29 23:28:42.808522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.806280 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 23:28:42.808522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.806569 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:42.824484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.811268 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:42.894520 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:42.863964 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:43.033093 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.027636 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:43.033093 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.027701 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:28:43.033093 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.027736 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:28:43.033093 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.027770 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:43.033093 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.027803 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:28:43.033093 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.027841 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:28:43.033093 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.027869 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:28:43.033093 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.027898 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:28:43.163486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.146695 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:43.163486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.152579 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 23:28:43.261579 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.261136 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:28:43.436542 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.431978 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:28:43.495241 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.479892 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:28:43.495241 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.491078 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:28:43.495241 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.492313 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:28:43.536588 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.504736 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:28:43.602365 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:43.602335 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:28:45.224795 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:28:45.224752 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:28:45.246241 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.246210 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh"] Oct 29 23:28:45.246845 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.246830 6507 cpu_manager.go:414] "containerMap: removing container" podUID="49ec50ce-e1cb-4201-9a60-147f1bcb1db8" containerName="revoke-keys" Oct 29 23:28:45.246898 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.246845 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ec50ce-e1cb-4201-9a60-147f1bcb1db8" containerName="revoke-keys" Oct 29 23:28:45.246898 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.246861 6507 cpu_manager.go:414] "containerMap: removing container" podUID="cda96310-6dec-41a2-8429-1751247210ab" containerName="revoke-keys" Oct 29 23:28:45.246898 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.246866 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cda96310-6dec-41a2-8429-1751247210ab" containerName="revoke-keys" Oct 29 23:28:45.246993 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.246980 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ec50ce-e1cb-4201-9a60-147f1bcb1db8" containerName="revoke-keys" Oct 29 23:28:45.246993 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.246991 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="cda96310-6dec-41a2-8429-1751247210ab" containerName="revoke-keys" Oct 29 23:28:45.250760 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.250743 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" Oct 29 23:28:45.272586 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.272547 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:28:45.272746 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.272685 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:28:45.272824 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.272763 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:28:45.273185 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.273167 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:28:45.276973 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.276947 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh"] Oct 29 23:28:45.317136 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.317097 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn9pj\" (UniqueName: \"kubernetes.io/projected/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-kube-api-access-fn9pj\") pod \"maas-api-revoke-sub-ae32e799-p29hh\" (UID: \"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" Oct 29 23:28:45.317354 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.317142 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae32e799-p29hh\" (UID: \"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" Oct 29 23:28:45.317416 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.317361 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae32e799-p29hh\" (UID: \"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" Oct 29 23:28:45.418206 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.418171 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae32e799-p29hh\" (UID: \"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" Oct 29 23:28:45.418385 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.418315 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn9pj\" (UniqueName: \"kubernetes.io/projected/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-kube-api-access-fn9pj\") pod \"maas-api-revoke-sub-ae32e799-p29hh\" (UID: \"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" Oct 29 23:28:45.418385 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.418351 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae32e799-p29hh\" (UID: \"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" Oct 29 23:28:45.418960 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.418937 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae32e799-p29hh\" (UID: \"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" Oct 29 23:28:45.420805 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.420786 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae32e799-p29hh\" (UID: \"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" Oct 29 23:28:45.435413 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.435383 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn9pj\" (UniqueName: \"kubernetes.io/projected/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-kube-api-access-fn9pj\") pod \"maas-api-revoke-sub-ae32e799-p29hh\" (UID: \"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" Oct 29 23:28:45.565519 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.565428 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" Oct 29 23:28:45.895182 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:45.895157 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh"] Oct 29 23:28:45.897546 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:28:45.897515 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a7ecc7_ee5f_4e7c_baa1_30758d4cbf28.slice/crio-810240defe32c791e3e77b2efbba2861af42b00c356e9ae13f992c6ef95e56f9 WatchSource:0}: Error finding container 810240defe32c791e3e77b2efbba2861af42b00c356e9ae13f992c6ef95e56f9: Status 404 returned error can't find the container with id 810240defe32c791e3e77b2efbba2861af42b00c356e9ae13f992c6ef95e56f9 Oct 29 23:28:46.659603 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:46.659502 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" event={"ID":"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28","Type":"ContainerStarted","Data":"ab3e5fe6c1c77678dc01b75efb38d8fd1f0a066fdc15801f71cc2c49d9736734"} Oct 29 23:28:46.659603 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:46.659572 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" event={"ID":"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28","Type":"ContainerStarted","Data":"810240defe32c791e3e77b2efbba2861af42b00c356e9ae13f992c6ef95e56f9"} Oct 29 23:28:46.696084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:46.696026 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" podStartSLOduration=1.696011162 podStartE2EDuration="1.696011162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:46.692048532 +0000 UTC m=+896.055053254" watchObservedRunningTime="2026-10-29 23:28:46.696011162 +0000 UTC m=+896.059015884" Oct 29 23:28:47.517319 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:47.517278 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9"] Oct 29 23:28:47.522249 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:47.522223 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" Oct 29 23:28:47.544775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:47.544743 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9"] Oct 29 23:28:47.555166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:47.555116 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c944a028-57d8-4711-a4ba-f581a6148bd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ba1c451-5b9s9\" (UID: \"c944a028-57d8-4711-a4ba-f581a6148bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" Oct 29 23:28:47.555343 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:47.555215 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c944a028-57d8-4711-a4ba-f581a6148bd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ba1c451-5b9s9\" (UID: \"c944a028-57d8-4711-a4ba-f581a6148bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" Oct 29 23:28:47.555423 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:47.555375 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wwdb\" (UniqueName: \"kubernetes.io/projected/c944a028-57d8-4711-a4ba-f581a6148bd8-kube-api-access-8wwdb\") pod \"maas-api-revoke-sub-3ba1c451-5b9s9\" (UID: \"c944a028-57d8-4711-a4ba-f581a6148bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" Oct 29 23:28:47.657111 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:47.657064 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c944a028-57d8-4711-a4ba-f581a6148bd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ba1c451-5b9s9\" (UID: \"c944a028-57d8-4711-a4ba-f581a6148bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" Oct 29 23:28:47.657318 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:47.657116 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c944a028-57d8-4711-a4ba-f581a6148bd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ba1c451-5b9s9\" (UID: \"c944a028-57d8-4711-a4ba-f581a6148bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" Oct 29 23:28:47.657318 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:47.657197 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wwdb\" (UniqueName: \"kubernetes.io/projected/c944a028-57d8-4711-a4ba-f581a6148bd8-kube-api-access-8wwdb\") pod \"maas-api-revoke-sub-3ba1c451-5b9s9\" (UID: \"c944a028-57d8-4711-a4ba-f581a6148bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" Oct 29 23:28:47.657803 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:47.657772 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c944a028-57d8-4711-a4ba-f581a6148bd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ba1c451-5b9s9\" (UID: \"c944a028-57d8-4711-a4ba-f581a6148bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" Oct 29 23:28:47.659497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:47.659473 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c944a028-57d8-4711-a4ba-f581a6148bd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ba1c451-5b9s9\" (UID: \"c944a028-57d8-4711-a4ba-f581a6148bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" Oct 29 23:28:47.668124 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:47.668093 6507 generic.go:356] "Generic (PLEG): container finished" podID="b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28" containerID="ab3e5fe6c1c77678dc01b75efb38d8fd1f0a066fdc15801f71cc2c49d9736734" exitCode=0 Oct 29 23:28:47.668477 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:47.668179 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" event={"ID":"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28","Type":"ContainerDied","Data":"ab3e5fe6c1c77678dc01b75efb38d8fd1f0a066fdc15801f71cc2c49d9736734"} Oct 29 23:28:47.674096 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:47.674073 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wwdb\" (UniqueName: \"kubernetes.io/projected/c944a028-57d8-4711-a4ba-f581a6148bd8-kube-api-access-8wwdb\") pod \"maas-api-revoke-sub-3ba1c451-5b9s9\" (UID: \"c944a028-57d8-4711-a4ba-f581a6148bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" Oct 29 23:28:47.835970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:47.835869 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" Oct 29 23:28:48.137062 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:48.137034 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9"] Oct 29 23:28:48.139389 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:28:48.139359 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc944a028_57d8_4711_a4ba_f581a6148bd8.slice/crio-2d207ad4ddef6797c34972f9574f25749533a6fc0d14d45f99a619d62443aecd WatchSource:0}: Error finding container 2d207ad4ddef6797c34972f9574f25749533a6fc0d14d45f99a619d62443aecd: Status 404 returned error can't find the container with id 2d207ad4ddef6797c34972f9574f25749533a6fc0d14d45f99a619d62443aecd Oct 29 23:28:48.771089 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:48.771048 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" event={"ID":"c944a028-57d8-4711-a4ba-f581a6148bd8","Type":"ContainerStarted","Data":"0c67e97d3ff7d369181c44001e236c3a2756a6dcaa02b231ce84fa9965e7f0d9"} Oct 29 23:28:48.771623 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:48.771102 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" event={"ID":"c944a028-57d8-4711-a4ba-f581a6148bd8","Type":"ContainerStarted","Data":"2d207ad4ddef6797c34972f9574f25749533a6fc0d14d45f99a619d62443aecd"} Oct 29 23:28:48.806248 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:48.806198 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" podStartSLOduration=1.806179543 podStartE2EDuration="1.806179543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:48.804597863 +0000 UTC m=+898.167602587" watchObservedRunningTime="2026-10-29 23:28:48.806179543 +0000 UTC m=+898.169184266" Oct 29 23:28:48.926802 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:48.926776 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" Oct 29 23:28:48.982420 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:48.982390 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-maas-api-xks-ca\") pod \"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28\" (UID: \"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28\") " Oct 29 23:28:48.982623 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:48.982501 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-kube-api-access-fn9pj\" (UniqueName: \"kubernetes.io/projected/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-kube-api-access-fn9pj\") pod \"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28\" (UID: \"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28\") " Oct 29 23:28:48.982623 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:48.982605 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-maas-api-openshift-ca\") pod \"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28\" (UID: \"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28\") " Oct 29 23:28:48.983400 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:48.983371 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-maas-api-openshift-ca" pod "b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28" (UID: "b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:48.986045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:48.986002 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-maas-api-xks-ca" pod "b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28" (UID: "b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:48.987828 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:48.987163 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-kube-api-access-fn9pj" pod "b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28" (UID: "b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28"). InnerVolumeSpecName "kube-api-access-fn9pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:49.084864 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:49.084784 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn9pj\" (UniqueName: \"kubernetes.io/projected/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-kube-api-access-fn9pj\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:28:49.084864 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:49.084814 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:28:49.084864 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:49.084824 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:28:49.779836 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:49.779797 6507 generic.go:356] "Generic (PLEG): container finished" podID="c944a028-57d8-4711-a4ba-f581a6148bd8" containerID="0c67e97d3ff7d369181c44001e236c3a2756a6dcaa02b231ce84fa9965e7f0d9" exitCode=0 Oct 29 23:28:49.780273 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:49.779883 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" event={"ID":"c944a028-57d8-4711-a4ba-f581a6148bd8","Type":"ContainerDied","Data":"0c67e97d3ff7d369181c44001e236c3a2756a6dcaa02b231ce84fa9965e7f0d9"} Oct 29 23:28:49.781142 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:49.781118 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" event={"ID":"b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28","Type":"ContainerDied","Data":"810240defe32c791e3e77b2efbba2861af42b00c356e9ae13f992c6ef95e56f9"} Oct 29 23:28:49.781142 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:49.781137 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh" Oct 29 23:28:49.781287 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:49.781159 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="810240defe32c791e3e77b2efbba2861af42b00c356e9ae13f992c6ef95e56f9" Oct 29 23:28:50.968819 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:50.968791 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" Oct 29 23:28:51.012252 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:51.012222 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c944a028-57d8-4711-a4ba-f581a6148bd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c944a028-57d8-4711-a4ba-f581a6148bd8-maas-api-openshift-ca\") pod \"c944a028-57d8-4711-a4ba-f581a6148bd8\" (UID: \"c944a028-57d8-4711-a4ba-f581a6148bd8\") " Oct 29 23:28:51.012549 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:51.012525 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c944a028-57d8-4711-a4ba-f581a6148bd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c944a028-57d8-4711-a4ba-f581a6148bd8-maas-api-xks-ca\") pod \"c944a028-57d8-4711-a4ba-f581a6148bd8\" (UID: \"c944a028-57d8-4711-a4ba-f581a6148bd8\") " Oct 29 23:28:51.012650 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:51.012632 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c944a028-57d8-4711-a4ba-f581a6148bd8-kube-api-access-8wwdb\" (UniqueName: \"kubernetes.io/projected/c944a028-57d8-4711-a4ba-f581a6148bd8-kube-api-access-8wwdb\") pod \"c944a028-57d8-4711-a4ba-f581a6148bd8\" (UID: \"c944a028-57d8-4711-a4ba-f581a6148bd8\") " Oct 29 23:28:51.012783 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:51.012629 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c944a028-57d8-4711-a4ba-f581a6148bd8-maas-api-openshift-ca" pod "c944a028-57d8-4711-a4ba-f581a6148bd8" (UID: "c944a028-57d8-4711-a4ba-f581a6148bd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:51.013130 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:51.013111 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c944a028-57d8-4711-a4ba-f581a6148bd8-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:28:51.023123 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:51.017637 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c944a028-57d8-4711-a4ba-f581a6148bd8-maas-api-xks-ca" pod "c944a028-57d8-4711-a4ba-f581a6148bd8" (UID: "c944a028-57d8-4711-a4ba-f581a6148bd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:51.023123 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:51.017663 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c944a028-57d8-4711-a4ba-f581a6148bd8-kube-api-access-8wwdb" pod "c944a028-57d8-4711-a4ba-f581a6148bd8" (UID: "c944a028-57d8-4711-a4ba-f581a6148bd8"). InnerVolumeSpecName "kube-api-access-8wwdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:51.114802 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:51.114720 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wwdb\" (UniqueName: \"kubernetes.io/projected/c944a028-57d8-4711-a4ba-f581a6148bd8-kube-api-access-8wwdb\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:28:51.114802 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:51.114758 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c944a028-57d8-4711-a4ba-f581a6148bd8-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:28:51.797739 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:51.797707 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" Oct 29 23:28:51.797920 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:51.797740 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9" event={"ID":"c944a028-57d8-4711-a4ba-f581a6148bd8","Type":"ContainerDied","Data":"2d207ad4ddef6797c34972f9574f25749533a6fc0d14d45f99a619d62443aecd"} Oct 29 23:28:51.797920 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:51.797787 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d207ad4ddef6797c34972f9574f25749533a6fc0d14d45f99a619d62443aecd" Oct 29 23:28:54.441944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.441911 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86"] Oct 29 23:28:54.442498 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.442484 6507 cpu_manager.go:414] "containerMap: removing container" podUID="c944a028-57d8-4711-a4ba-f581a6148bd8" containerName="revoke-keys" Oct 29 23:28:54.442553 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.442499 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c944a028-57d8-4711-a4ba-f581a6148bd8" containerName="revoke-keys" Oct 29 23:28:54.442553 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.442536 6507 cpu_manager.go:414] "containerMap: removing container" podUID="b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28" containerName="revoke-keys" Oct 29 23:28:54.442553 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.442541 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28" containerName="revoke-keys" Oct 29 23:28:54.442644 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.442636 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28" containerName="revoke-keys" Oct 29 23:28:54.442677 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.442650 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="c944a028-57d8-4711-a4ba-f581a6148bd8" containerName="revoke-keys" Oct 29 23:28:54.447867 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.447846 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.478808 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.478774 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86"] Oct 29 23:28:54.565440 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.565389 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-envoy\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.565665 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.565485 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-workload-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.565665 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.565582 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-credential-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.565665 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.565616 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqnv8\" (UniqueName: \"kubernetes.io/projected/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-kube-api-access-jqnv8\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.565791 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.565703 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-podinfo\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.565791 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.565770 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-token\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.565864 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.565837 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-workload-certs\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.565902 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.565862 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istiod-ca-cert\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.566050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.566025 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-data\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.575878 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.575838 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw"] Oct 29 23:28:54.580254 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.580230 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.605290 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.605256 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw"] Oct 29 23:28:54.741903 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.741803 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-data\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.742169 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.742132 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150ee23d-c37d-433f-812d-e11da74045c3-istiod-ca-cert\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.742298 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.742176 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzw8j\" (UniqueName: \"kubernetes.io/projected/150ee23d-c37d-433f-812d-e11da74045c3-kube-api-access-rzw8j\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.742298 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.742223 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-envoy\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.742298 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.742263 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-workload-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.742543 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.742302 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150ee23d-c37d-433f-812d-e11da74045c3-istio-podinfo\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.742543 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.742353 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-credential-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.742543 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.742381 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqnv8\" (UniqueName: \"kubernetes.io/projected/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-kube-api-access-jqnv8\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.742543 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.742415 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-istio-envoy\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.742543 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.742446 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-istio-data\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.742543 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.742528 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-workload-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.742900 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.742702 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-podinfo\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.742900 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.742862 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150ee23d-c37d-433f-812d-e11da74045c3-istio-token\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.743033 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.742908 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-token\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.743033 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.742932 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-credential-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.743033 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.742968 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-workload-certs\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.743033 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.743028 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-workload-certs\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.743234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.743053 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istiod-ca-cert\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.747048 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.746949 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-credential-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.747264 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.747239 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-workload-socket\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.748439 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.748413 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-workload-certs\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.748654 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.748627 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istiod-ca-cert\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.748760 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.748737 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-data\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.761781 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.761749 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-envoy\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.762719 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.762677 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-podinfo\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.765363 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.765332 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-token\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.770771 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.770725 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqnv8\" (UniqueName: \"kubernetes.io/projected/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-kube-api-access-jqnv8\") pod \"e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:54.844963 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.844928 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-istio-envoy\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.845155 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.844970 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-istio-data\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.845155 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.845015 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-workload-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.845155 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.845107 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150ee23d-c37d-433f-812d-e11da74045c3-istio-token\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.845155 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.845137 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-credential-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.845392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.845166 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-workload-certs\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.845392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.845352 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150ee23d-c37d-433f-812d-e11da74045c3-istiod-ca-cert\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.845392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.845381 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzw8j\" (UniqueName: \"kubernetes.io/projected/150ee23d-c37d-433f-812d-e11da74045c3-kube-api-access-rzw8j\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.845570 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.845397 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-istio-data\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.845570 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.845439 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150ee23d-c37d-433f-812d-e11da74045c3-istio-podinfo\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.845570 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.845500 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-workload-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.845733 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.845692 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-workload-certs\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.845787 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.845730 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-credential-socket\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.846372 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.846344 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150ee23d-c37d-433f-812d-e11da74045c3-istiod-ca-cert\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.847901 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.847878 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-istio-envoy\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.848149 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.848129 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150ee23d-c37d-433f-812d-e11da74045c3-istio-podinfo\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.862041 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.862012 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150ee23d-c37d-433f-812d-e11da74045c3-istio-token\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.862187 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.862048 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzw8j\" (UniqueName: \"kubernetes.io/projected/150ee23d-c37d-433f-812d-e11da74045c3-kube-api-access-rzw8j\") pod \"e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:54.895479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:54.895430 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:55.063574 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:55.063473 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:55.222212 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:55.222184 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw"] Oct 29 23:28:55.223184 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:28:55.223157 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150ee23d_c37d_433f_812d_e11da74045c3.slice/crio-9df94f84fa6eec1886492866983e222b9429d3310940717e7543c30f72bd35c8 WatchSource:0}: Error finding container 9df94f84fa6eec1886492866983e222b9429d3310940717e7543c30f72bd35c8: Status 404 returned error can't find the container with id 9df94f84fa6eec1886492866983e222b9429d3310940717e7543c30f72bd35c8 Oct 29 23:28:55.225769 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:55.225737 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:28:55.225846 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:55.225803 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:28:55.225846 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:55.225830 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:28:55.371225 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:55.371193 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86"] Oct 29 23:28:55.372566 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:28:55.372538 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d1d7e4_9a92_4ffa_9cb8_20dee7904a50.slice/crio-0f666c9ed0f4a2ce7663173fba4c9ba6c03f6c057d3b9d5a7e9ca4e13480fccf WatchSource:0}: Error finding container 0f666c9ed0f4a2ce7663173fba4c9ba6c03f6c057d3b9d5a7e9ca4e13480fccf: Status 404 returned error can't find the container with id 0f666c9ed0f4a2ce7663173fba4c9ba6c03f6c057d3b9d5a7e9ca4e13480fccf Oct 29 23:28:55.375419 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:55.375383 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:28:55.375535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:55.375452 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:28:55.375535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:55.375516 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:28:55.838320 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:55.838282 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" event={"ID":"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50","Type":"ContainerStarted","Data":"6397bdc304d6a63a78bdcd971dc5c03b09f522a189a93e851b1e92c7e0c9813a"} Oct 29 23:28:55.838779 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:55.838366 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" event={"ID":"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50","Type":"ContainerStarted","Data":"0f666c9ed0f4a2ce7663173fba4c9ba6c03f6c057d3b9d5a7e9ca4e13480fccf"} Oct 29 23:28:55.841401 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:55.841370 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" event={"ID":"150ee23d-c37d-433f-812d-e11da74045c3","Type":"ContainerStarted","Data":"b1c52ed5f836a3c68b061c1373319be0d2549f0e5a4553f63f58f8e6cde9d646"} Oct 29 23:28:55.841533 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:55.841427 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" event={"ID":"150ee23d-c37d-433f-812d-e11da74045c3","Type":"ContainerStarted","Data":"9df94f84fa6eec1886492866983e222b9429d3310940717e7543c30f72bd35c8"} Oct 29 23:28:55.879285 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:55.879095 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" podStartSLOduration=1.879078611 podStartE2EDuration="1.879078611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:55.877245572 +0000 UTC m=+905.240250308" watchObservedRunningTime="2026-10-29 23:28:55.879078611 +0000 UTC m=+905.242083334" Oct 29 23:28:55.895612 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:55.895583 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:28:55.934607 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:55.932385 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" podStartSLOduration=1.9323629279999999 podStartE2EDuration="1.932362928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:55.9176805 +0000 UTC m=+905.280685224" watchObservedRunningTime="2026-10-29 23:28:55.932362928 +0000 UTC m=+905.295367653" Oct 29 23:28:56.064005 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.063966 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:28:56.065170 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.065143 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 29 23:28:56.065334 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.065313 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" podUID="77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 29 23:28:56.595947 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.595910 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w"] Oct 29 23:28:56.600722 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.600699 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" Oct 29 23:28:56.645479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.637013 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:28:56.645479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.637155 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:28:56.645479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.637185 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:28:56.645479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.640552 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:28:56.663616 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.663480 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w"] Oct 29 23:28:56.680922 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.680835 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10904812-7716-478e-95c0-796a65c4d269-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50a539e3-dz55w\" (UID: \"10904812-7716-478e-95c0-796a65c4d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" Oct 29 23:28:56.680922 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.680903 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l4pv\" (UniqueName: \"kubernetes.io/projected/10904812-7716-478e-95c0-796a65c4d269-kube-api-access-8l4pv\") pod \"maas-api-revoke-sub-50a539e3-dz55w\" (UID: \"10904812-7716-478e-95c0-796a65c4d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" Oct 29 23:28:56.681093 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.681030 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10904812-7716-478e-95c0-796a65c4d269-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50a539e3-dz55w\" (UID: \"10904812-7716-478e-95c0-796a65c4d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" Oct 29 23:28:56.782105 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.782042 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l4pv\" (UniqueName: \"kubernetes.io/projected/10904812-7716-478e-95c0-796a65c4d269-kube-api-access-8l4pv\") pod \"maas-api-revoke-sub-50a539e3-dz55w\" (UID: \"10904812-7716-478e-95c0-796a65c4d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" Oct 29 23:28:56.782274 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.782136 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10904812-7716-478e-95c0-796a65c4d269-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50a539e3-dz55w\" (UID: \"10904812-7716-478e-95c0-796a65c4d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" Oct 29 23:28:56.782274 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.782254 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10904812-7716-478e-95c0-796a65c4d269-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50a539e3-dz55w\" (UID: \"10904812-7716-478e-95c0-796a65c4d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" Oct 29 23:28:56.782903 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.782883 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10904812-7716-478e-95c0-796a65c4d269-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50a539e3-dz55w\" (UID: \"10904812-7716-478e-95c0-796a65c4d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" Oct 29 23:28:56.787012 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.784714 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10904812-7716-478e-95c0-796a65c4d269-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50a539e3-dz55w\" (UID: \"10904812-7716-478e-95c0-796a65c4d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" Oct 29 23:28:56.799445 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.799416 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l4pv\" (UniqueName: \"kubernetes.io/projected/10904812-7716-478e-95c0-796a65c4d269-kube-api-access-8l4pv\") pod \"maas-api-revoke-sub-50a539e3-dz55w\" (UID: \"10904812-7716-478e-95c0-796a65c4d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" Oct 29 23:28:56.897616 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.897529 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 23:28:56.897616 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.897586 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" podUID="150ee23d-c37d-433f-812d-e11da74045c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 23:28:56.975332 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:56.975296 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" Oct 29 23:28:57.064964 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:57.064922 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 29 23:28:57.065142 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:57.064986 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" podUID="77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 29 23:28:57.355485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:57.355324 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w"] Oct 29 23:28:57.356606 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:28:57.356563 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10904812_7716_478e_95c0_796a65c4d269.slice/crio-a8dfa44e268c510f426712d5bcd16adab2d49362a6b300b175ff0edbec63b808 WatchSource:0}: Error finding container a8dfa44e268c510f426712d5bcd16adab2d49362a6b300b175ff0edbec63b808: Status 404 returned error can't find the container with id a8dfa44e268c510f426712d5bcd16adab2d49362a6b300b175ff0edbec63b808 Oct 29 23:28:57.870712 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:57.870665 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" event={"ID":"10904812-7716-478e-95c0-796a65c4d269","Type":"ContainerStarted","Data":"c5444b51cda3df973a081208848f7b551c7d5dcf31eea78439d42585c0edd661"} Oct 29 23:28:57.870883 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:57.870736 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" event={"ID":"10904812-7716-478e-95c0-796a65c4d269","Type":"ContainerStarted","Data":"a8dfa44e268c510f426712d5bcd16adab2d49362a6b300b175ff0edbec63b808"} Oct 29 23:28:57.896382 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:57.896353 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 23:28:57.896544 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:57.896402 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" podUID="150ee23d-c37d-433f-812d-e11da74045c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 23:28:57.929975 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:57.929909 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" podStartSLOduration=1.929887017 podStartE2EDuration="1.929887017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:57.926649862 +0000 UTC m=+907.289654588" watchObservedRunningTime="2026-10-29 23:28:57.929887017 +0000 UTC m=+907.292891742" Oct 29 23:28:58.064034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:58.063934 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 29 23:28:58.064034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:58.063995 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" podUID="77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 29 23:28:58.878432 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:58.878398 6507 generic.go:356] "Generic (PLEG): container finished" podID="10904812-7716-478e-95c0-796a65c4d269" containerID="c5444b51cda3df973a081208848f7b551c7d5dcf31eea78439d42585c0edd661" exitCode=0 Oct 29 23:28:58.878617 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:58.878512 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" event={"ID":"10904812-7716-478e-95c0-796a65c4d269","Type":"ContainerDied","Data":"c5444b51cda3df973a081208848f7b551c7d5dcf31eea78439d42585c0edd661"} Oct 29 23:28:58.896010 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:58.895977 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 23:28:58.896128 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:58.896047 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" podUID="150ee23d-c37d-433f-812d-e11da74045c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 23:28:59.064329 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:59.064287 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 29 23:28:59.064718 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:59.064384 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" podUID="77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 29 23:28:59.896061 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:59.896021 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 23:28:59.896219 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:28:59.896089 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" podUID="150ee23d-c37d-433f-812d-e11da74045c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 23:29:00.064281 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.064110 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 29 23:29:00.064281 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.064160 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" podUID="77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 29 23:29:00.356497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.355926 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" Oct 29 23:29:00.424497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.417157 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10904812-7716-478e-95c0-796a65c4d269-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10904812-7716-478e-95c0-796a65c4d269-maas-api-openshift-ca\") pod \"10904812-7716-478e-95c0-796a65c4d269\" (UID: \"10904812-7716-478e-95c0-796a65c4d269\") " Oct 29 23:29:00.424497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.417376 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10904812-7716-478e-95c0-796a65c4d269-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10904812-7716-478e-95c0-796a65c4d269-maas-api-xks-ca\") pod \"10904812-7716-478e-95c0-796a65c4d269\" (UID: \"10904812-7716-478e-95c0-796a65c4d269\") " Oct 29 23:29:00.424497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.417426 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10904812-7716-478e-95c0-796a65c4d269-kube-api-access-8l4pv\" (UniqueName: \"kubernetes.io/projected/10904812-7716-478e-95c0-796a65c4d269-kube-api-access-8l4pv\") pod \"10904812-7716-478e-95c0-796a65c4d269\" (UID: \"10904812-7716-478e-95c0-796a65c4d269\") " Oct 29 23:29:00.437484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.435880 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10904812-7716-478e-95c0-796a65c4d269-maas-api-openshift-ca" pod "10904812-7716-478e-95c0-796a65c4d269" (UID: "10904812-7716-478e-95c0-796a65c4d269"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:00.457354 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.449995 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10904812-7716-478e-95c0-796a65c4d269-maas-api-xks-ca" pod "10904812-7716-478e-95c0-796a65c4d269" (UID: "10904812-7716-478e-95c0-796a65c4d269"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:00.476203 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.475805 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10904812-7716-478e-95c0-796a65c4d269-kube-api-access-8l4pv" pod "10904812-7716-478e-95c0-796a65c4d269" (UID: "10904812-7716-478e-95c0-796a65c4d269"). InnerVolumeSpecName "kube-api-access-8l4pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:00.550923 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.550829 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10904812-7716-478e-95c0-796a65c4d269-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:00.551132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.551120 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l4pv\" (UniqueName: \"kubernetes.io/projected/10904812-7716-478e-95c0-796a65c4d269-kube-api-access-8l4pv\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:00.551203 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.551194 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10904812-7716-478e-95c0-796a65c4d269-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:00.895625 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.895587 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" Oct 29 23:29:00.895625 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.895608 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w" event={"ID":"10904812-7716-478e-95c0-796a65c4d269","Type":"ContainerDied","Data":"a8dfa44e268c510f426712d5bcd16adab2d49362a6b300b175ff0edbec63b808"} Oct 29 23:29:00.895838 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.895667 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8dfa44e268c510f426712d5bcd16adab2d49362a6b300b175ff0edbec63b808" Oct 29 23:29:00.895838 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.895739 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 23:29:00.895838 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:00.895795 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" podUID="150ee23d-c37d-433f-812d-e11da74045c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 23:29:01.065055 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.064960 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 29 23:29:01.065055 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.065019 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" podUID="77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 29 23:29:01.831967 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.831925 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn"] Oct 29 23:29:01.832640 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.832623 6507 cpu_manager.go:414] "containerMap: removing container" podUID="10904812-7716-478e-95c0-796a65c4d269" containerName="revoke-keys" Oct 29 23:29:01.832709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.832641 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10904812-7716-478e-95c0-796a65c4d269" containerName="revoke-keys" Oct 29 23:29:01.832766 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.832757 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="10904812-7716-478e-95c0-796a65c4d269" containerName="revoke-keys" Oct 29 23:29:01.836660 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.836636 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn" Oct 29 23:29:01.856526 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.856486 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn"] Oct 29 23:29:01.859867 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.859840 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:29:01.860063 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.859881 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:29:01.860063 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.860054 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:29:01.860262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.860008 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:29:01.899860 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.899823 6507 patch_prober.go:29] interesting pod/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 23:29:01.900229 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.900191 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" podUID="150ee23d-c37d-433f-812d-e11da74045c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 23:29:01.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.973016 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8e6875-435c-4ae7-830e-bbabe94589c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd375e44-m78nn\" (UID: \"4e8e6875-435c-4ae7-830e-bbabe94589c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn" Oct 29 23:29:01.973050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.973052 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8e6875-435c-4ae7-830e-bbabe94589c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd375e44-m78nn\" (UID: \"4e8e6875-435c-4ae7-830e-bbabe94589c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn" Oct 29 23:29:01.973391 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:01.973202 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbnmx\" (UniqueName: \"kubernetes.io/projected/4e8e6875-435c-4ae7-830e-bbabe94589c0-kube-api-access-lbnmx\") pod \"maas-api-revoke-sub-dd375e44-m78nn\" (UID: \"4e8e6875-435c-4ae7-830e-bbabe94589c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn" Oct 29 23:29:02.064384 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:02.064350 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 29 23:29:02.064585 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:02.064405 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" podUID="77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 29 23:29:02.075177 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:02.075139 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8e6875-435c-4ae7-830e-bbabe94589c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd375e44-m78nn\" (UID: \"4e8e6875-435c-4ae7-830e-bbabe94589c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn" Oct 29 23:29:02.075653 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:02.075183 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8e6875-435c-4ae7-830e-bbabe94589c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd375e44-m78nn\" (UID: \"4e8e6875-435c-4ae7-830e-bbabe94589c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn" Oct 29 23:29:02.075653 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:02.075293 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbnmx\" (UniqueName: \"kubernetes.io/projected/4e8e6875-435c-4ae7-830e-bbabe94589c0-kube-api-access-lbnmx\") pod \"maas-api-revoke-sub-dd375e44-m78nn\" (UID: \"4e8e6875-435c-4ae7-830e-bbabe94589c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn" Oct 29 23:29:02.075924 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:02.075892 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8e6875-435c-4ae7-830e-bbabe94589c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd375e44-m78nn\" (UID: \"4e8e6875-435c-4ae7-830e-bbabe94589c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn" Oct 29 23:29:02.077840 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:02.077814 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8e6875-435c-4ae7-830e-bbabe94589c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd375e44-m78nn\" (UID: \"4e8e6875-435c-4ae7-830e-bbabe94589c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn" Oct 29 23:29:02.092253 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:02.092176 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbnmx\" (UniqueName: \"kubernetes.io/projected/4e8e6875-435c-4ae7-830e-bbabe94589c0-kube-api-access-lbnmx\") pod \"maas-api-revoke-sub-dd375e44-m78nn\" (UID: \"4e8e6875-435c-4ae7-830e-bbabe94589c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn" Oct 29 23:29:02.153654 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:02.153610 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn" Oct 29 23:29:02.589378 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:02.589341 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn"] Oct 29 23:29:02.598757 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:29:02.598677 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8e6875_435c_4ae7_830e_bbabe94589c0.slice/crio-4ae74ed107ca39539a7f5ea13407ba43eb3a53b9df08f7f0cbbb95e66123deea WatchSource:0}: Error finding container 4ae74ed107ca39539a7f5ea13407ba43eb3a53b9df08f7f0cbbb95e66123deea: Status 404 returned error can't find the container with id 4ae74ed107ca39539a7f5ea13407ba43eb3a53b9df08f7f0cbbb95e66123deea Oct 29 23:29:02.942237 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:02.942184 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn" event={"ID":"4e8e6875-435c-4ae7-830e-bbabe94589c0","Type":"ContainerStarted","Data":"4ae74ed107ca39539a7f5ea13407ba43eb3a53b9df08f7f0cbbb95e66123deea"} Oct 29 23:29:03.117134 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:03.117006 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:29:03.118442 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:03.118422 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:29:03.119321 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:03.119298 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:29:03.206528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:03.206405 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:29:03.206928 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:03.206898 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:29:03.210190 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:03.208270 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:29:03.951308 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:03.951269 6507 generic.go:356] "Generic (PLEG): container finished" podID="4e8e6875-435c-4ae7-830e-bbabe94589c0" containerID="00ba596edceac4f7cd5e8df8d746a8e4b8adef4ca0117c732f3c39b924019f0a" exitCode=0 Oct 29 23:29:03.951541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:03.951344 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn" event={"ID":"4e8e6875-435c-4ae7-830e-bbabe94589c0","Type":"ContainerDied","Data":"00ba596edceac4f7cd5e8df8d746a8e4b8adef4ca0117c732f3c39b924019f0a"} Oct 29 23:29:05.152375 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:05.152343 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn" Oct 29 23:29:05.245194 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:05.244927 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8e6875-435c-4ae7-830e-bbabe94589c0-kube-api-access-lbnmx\" (UniqueName: \"kubernetes.io/projected/4e8e6875-435c-4ae7-830e-bbabe94589c0-kube-api-access-lbnmx\") pod \"4e8e6875-435c-4ae7-830e-bbabe94589c0\" (UID: \"4e8e6875-435c-4ae7-830e-bbabe94589c0\") " Oct 29 23:29:05.245194 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:05.245119 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e8e6875-435c-4ae7-830e-bbabe94589c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8e6875-435c-4ae7-830e-bbabe94589c0-maas-api-openshift-ca\") pod \"4e8e6875-435c-4ae7-830e-bbabe94589c0\" (UID: \"4e8e6875-435c-4ae7-830e-bbabe94589c0\") " Oct 29 23:29:05.245194 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:05.245181 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e8e6875-435c-4ae7-830e-bbabe94589c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8e6875-435c-4ae7-830e-bbabe94589c0-maas-api-xks-ca\") pod \"4e8e6875-435c-4ae7-830e-bbabe94589c0\" (UID: \"4e8e6875-435c-4ae7-830e-bbabe94589c0\") " Oct 29 23:29:05.270480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:05.270162 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8e6875-435c-4ae7-830e-bbabe94589c0-maas-api-openshift-ca" pod "4e8e6875-435c-4ae7-830e-bbabe94589c0" (UID: "4e8e6875-435c-4ae7-830e-bbabe94589c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:05.270667 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:05.270610 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e8e6875-435c-4ae7-830e-bbabe94589c0-maas-api-xks-ca" pod "4e8e6875-435c-4ae7-830e-bbabe94589c0" (UID: "4e8e6875-435c-4ae7-830e-bbabe94589c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:05.277285 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:05.276941 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8e6875-435c-4ae7-830e-bbabe94589c0-kube-api-access-lbnmx" pod "4e8e6875-435c-4ae7-830e-bbabe94589c0" (UID: "4e8e6875-435c-4ae7-830e-bbabe94589c0"). InnerVolumeSpecName "kube-api-access-lbnmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:05.347176 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:05.347136 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbnmx\" (UniqueName: \"kubernetes.io/projected/4e8e6875-435c-4ae7-830e-bbabe94589c0-kube-api-access-lbnmx\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:05.347176 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:05.347170 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8e6875-435c-4ae7-830e-bbabe94589c0-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:05.347176 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:05.347181 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8e6875-435c-4ae7-830e-bbabe94589c0-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:05.970042 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:05.970010 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn" Oct 29 23:29:05.970257 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:05.970040 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn" event={"ID":"4e8e6875-435c-4ae7-830e-bbabe94589c0","Type":"ContainerDied","Data":"4ae74ed107ca39539a7f5ea13407ba43eb3a53b9df08f7f0cbbb95e66123deea"} Oct 29 23:29:05.970257 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:05.970121 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ae74ed107ca39539a7f5ea13407ba43eb3a53b9df08f7f0cbbb95e66123deea" Oct 29 23:29:07.498485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:07.497578 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:07.640484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:07.639545 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded" Oct 29 23:29:07.858551 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:07.857600 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:08.156551 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:08.155593 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 29 23:29:08.507486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:08.504723 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:08.507486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:08.505201 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:29:08.507486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:08.505610 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 23:29:08.507486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:08.505875 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 23:29:08.507486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:08.506158 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:29:08.559515 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:08.557558 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:08.615482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:08.612550 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 29 23:29:08.766561 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:08.652589 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:08.793642 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:08.792555 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:08.793642 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:08.793342 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:08.879480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:08.858485 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:08.917194 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:08.915041 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:29:09.041582 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:09.041367 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:09.178547 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:09.157695 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 29 23:29:09.178547 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:09.158060 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:29:09.314536 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:09.275951 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:09.555581 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:09.555518 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:09.567209 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:09.567106 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:29:09.568485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:09.568436 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 23:29:09.569698 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:09.569678 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:29:09.569882 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:09.569867 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:29:09.617483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:09.616594 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 29 23:29:09.617483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:09.616912 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:29:10.045244 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:10.045192 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 29 23:29:10.101960 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:10.101915 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 23:29:10.113771 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:10.113731 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:29:10.168579 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:10.140100 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 29 23:29:10.249550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:10.249494 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 29 23:29:10.259836 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:10.259777 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 29 23:29:10.260166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:10.260142 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 29 23:29:10.373397 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:10.314848 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:29:10.374947 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:10.374923 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:29:10.427553 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:10.340819 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:10.428549 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:10.398743 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:29:10.428767 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:10.428732 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" podUID="aa280316-fb6c-40db-8658-65b84bb4382b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:10.435826 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:10.435798 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:29:10.549487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:10.533805 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:10.779482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:10.777082 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:29:10.832484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:10.805046 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 23:29:11.080681 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.080563 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded" Oct 29 23:29:11.232539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.186567 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:11.233108 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.209600 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 23:29:11.233530 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.213106 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:11.233893 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.213501 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 29 23:29:11.261895 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.238553 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:11.301608 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.301556 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 29 23:29:11.361095 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.360993 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:11.432214 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.432171 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 23:29:11.453272 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.451324 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:29:11.453272 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.451400 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:29:11.453272 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.451437 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:29:11.507496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.507451 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv"] Oct 29 23:29:11.508686 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.508661 6507 cpu_manager.go:414] "containerMap: removing container" podUID="4e8e6875-435c-4ae7-830e-bbabe94589c0" containerName="revoke-keys" Oct 29 23:29:11.546762 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.544361 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8e6875-435c-4ae7-830e-bbabe94589c0" containerName="revoke-keys" Oct 29 23:29:11.547237 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.547210 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8e6875-435c-4ae7-830e-bbabe94589c0" containerName="revoke-keys" Oct 29 23:29:11.574132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.574091 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv"] Oct 29 23:29:11.574423 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.574397 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5"] Oct 29 23:29:11.574674 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.574640 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.710692 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.710642 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-podinfo\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.711033 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.710994 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-workload-certs\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.711393 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.711373 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-token\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.711659 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.711609 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-envoy\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.711859 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.711836 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-workload-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.712064 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.712028 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-data\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.712259 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.712241 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/089367aa-c3e8-4f67-af00-697e5d7555bd-istiod-ca-cert\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.712422 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.712403 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-credential-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.712801 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.712757 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bfk6\" (UniqueName: \"kubernetes.io/projected/089367aa-c3e8-4f67-af00-697e5d7555bd-kube-api-access-5bfk6\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.716347 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.716315 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5"] Oct 29 23:29:11.716578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.716545 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6"] Oct 29 23:29:11.718637 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.718618 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:11.746475 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.746428 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf"] Oct 29 23:29:11.757893 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.757677 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:11.759636 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.759608 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6"] Oct 29 23:29:11.768853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.759880 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf"] Oct 29 23:29:11.768853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.761032 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:11.771836 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.769387 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:29:11.814925 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.814876 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:11.819677 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.819630 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-workload-certs\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.834321 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.833643 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-workload-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:11.835272 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.835244 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/039b4caf-d11b-4933-af15-2efb408c36e3-istiod-ca-cert\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:11.835513 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.835492 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istiod-ca-cert\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:11.835683 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.835665 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/039b4caf-d11b-4933-af15-2efb408c36e3-istio-token\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:11.835821 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.835803 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpm4g\" (UniqueName: \"kubernetes.io/projected/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-kube-api-access-fpm4g\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:11.836046 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.836009 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-istio-envoy\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:11.836208 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.836190 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-token\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.836618 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.836598 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/039b4caf-d11b-4933-af15-2efb408c36e3-istio-podinfo\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:11.836785 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.836766 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-envoy\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.836925 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.836909 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-workload-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.837054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.837039 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-data\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.837202 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.837185 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/089367aa-c3e8-4f67-af00-697e5d7555bd-istiod-ca-cert\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.837325 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.837306 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-credential-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.837487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.837449 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8b29\" (UniqueName: \"kubernetes.io/projected/039b4caf-d11b-4933-af15-2efb408c36e3-kube-api-access-v8b29\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:11.837645 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.837627 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-data\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:11.837837 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.837819 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-token\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:11.837977 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.837960 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-workload-certs\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:11.838099 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.838082 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-istio-data\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:11.838307 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.838288 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-credential-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:11.838429 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.838412 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-podinfo\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:11.838614 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.838594 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bfk6\" (UniqueName: \"kubernetes.io/projected/089367aa-c3e8-4f67-af00-697e5d7555bd-kube-api-access-5bfk6\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.838748 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.838731 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-credential-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:11.838871 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.838843 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-workload-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.855236 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.838884 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-workload-certs\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:11.855565 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.855539 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-podinfo\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.855715 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.855700 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-envoy\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:11.855849 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.855836 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-workload-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:11.855992 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.855969 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:29:11.856082 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.839217 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-data\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.860516 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.840047 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/089367aa-c3e8-4f67-af00-697e5d7555bd-istiod-ca-cert\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.867481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.839722 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-credential-socket\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.867745 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.826439 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-workload-certs\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.867860 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.863805 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:29:11.881486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.880348 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:29:11.891495 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.883969 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-envoy\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.911221 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.911188 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-token\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.929817 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.920974 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bfk6\" (UniqueName: \"kubernetes.io/projected/089367aa-c3e8-4f67-af00-697e5d7555bd-kube-api-access-5bfk6\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.940657 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.930787 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-podinfo\") pod \"e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:11.979079 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:11.978995 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:29:12.018493 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.018029 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8b29\" (UniqueName: \"kubernetes.io/projected/039b4caf-d11b-4933-af15-2efb408c36e3-kube-api-access-v8b29\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.018493 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.018101 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6451f721-f9ac-4445-b719-bd6e21820d72-istiod-ca-cert\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.021550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.018813 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-data\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.021550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.019113 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-token\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.021550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.019185 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-workload-certs\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.021550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.019514 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-istio-data\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.021550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.019641 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-credential-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.021550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.019678 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-podinfo\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.021550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.020430 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-credential-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.021550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.020888 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-workload-certs\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.021550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.020948 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-istio-data\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.022132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.021023 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-workload-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.022132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.021744 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-workload-certs\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.022132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.021828 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-envoy\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.022132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.021930 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-workload-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.022132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.021994 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-workload-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.022132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.022050 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/039b4caf-d11b-4933-af15-2efb408c36e3-istiod-ca-cert\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.022132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.022121 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istiod-ca-cert\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.022503 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.022182 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/039b4caf-d11b-4933-af15-2efb408c36e3-istio-token\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.022503 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.022315 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpm4g\" (UniqueName: \"kubernetes.io/projected/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-kube-api-access-fpm4g\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.022503 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.022376 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-credential-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.022503 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.022438 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6451f721-f9ac-4445-b719-bd6e21820d72-istio-token\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.022718 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.022592 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-istio-envoy\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.022718 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.022655 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6451f721-f9ac-4445-b719-bd6e21820d72-istio-podinfo\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.022812 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.022771 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/039b4caf-d11b-4933-af15-2efb408c36e3-istio-podinfo\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.022864 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.022823 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvgxk\" (UniqueName: \"kubernetes.io/projected/6451f721-f9ac-4445-b719-bd6e21820d72-kube-api-access-fvgxk\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.032483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.022952 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-istio-envoy\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.036431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.034451 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-data\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.050222 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.047796 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-workload-certs\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.050222 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.048348 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-istio-data\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.050222 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.048623 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-credential-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.052712 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.052688 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:29:12.081679 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.068293 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:12.086746 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.084276 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-credential-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.086746 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.084973 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-workload-certs\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.089603 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.088526 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-envoy\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.089603 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.088980 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-workload-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.089603 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.089365 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-workload-socket\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.090219 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.090193 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/039b4caf-d11b-4933-af15-2efb408c36e3-istiod-ca-cert\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.091092 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.091065 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:29:12.091234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.091208 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istiod-ca-cert\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.091795 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.091765 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-podinfo\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.094269 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.094243 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-token\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.094824 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.094800 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:29:12.094824 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.094813 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8b29\" (UniqueName: \"kubernetes.io/projected/039b4caf-d11b-4933-af15-2efb408c36e3-kube-api-access-v8b29\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.101821 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.099236 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:29:12.104598 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.104572 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-istio-envoy\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.107599 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.107575 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/039b4caf-d11b-4933-af15-2efb408c36e3-istio-podinfo\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.109746 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.109725 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/039b4caf-d11b-4933-af15-2efb408c36e3-istio-token\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.111200 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.111177 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpm4g\" (UniqueName: \"kubernetes.io/projected/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-kube-api-access-fpm4g\") pod \"e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.125272 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.125231 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-credential-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.125501 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.125301 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6451f721-f9ac-4445-b719-bd6e21820d72-istio-token\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.125501 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.125374 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6451f721-f9ac-4445-b719-bd6e21820d72-istio-podinfo\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.125501 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.125415 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvgxk\" (UniqueName: \"kubernetes.io/projected/6451f721-f9ac-4445-b719-bd6e21820d72-kube-api-access-fvgxk\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.125785 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.125507 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-istio-envoy\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.125785 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.125559 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6451f721-f9ac-4445-b719-bd6e21820d72-istiod-ca-cert\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.125785 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.125697 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-workload-certs\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.125785 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.125724 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-istio-data\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.125785 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.125761 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-workload-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.126130 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.126109 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-credential-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.126168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.126153 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-workload-socket\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.127544 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.126785 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-istio-data\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.127641 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.126914 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6451f721-f9ac-4445-b719-bd6e21820d72-istiod-ca-cert\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.127747 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.127718 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-workload-certs\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.128684 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.128660 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-istio-envoy\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.133401 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.133372 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6451f721-f9ac-4445-b719-bd6e21820d72-istio-podinfo\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.142424 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.142393 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6451f721-f9ac-4445-b719-bd6e21820d72-istio-token\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.143160 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.143134 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvgxk\" (UniqueName: \"kubernetes.io/projected/6451f721-f9ac-4445-b719-bd6e21820d72-kube-api-access-fvgxk\") pod \"e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.175104 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.174971 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:12.400928 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.400893 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:12.410971 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.410926 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:12.418338 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.418312 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv"] Oct 29 23:29:12.419004 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:29:12.418974 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089367aa_c3e8_4f67_af00_697e5d7555bd.slice/crio-8b882730a80fc50a4bfa0ed274b007d29245882d22fa74248396ef377799978a WatchSource:0}: Error finding container 8b882730a80fc50a4bfa0ed274b007d29245882d22fa74248396ef377799978a: Status 404 returned error can't find the container with id 8b882730a80fc50a4bfa0ed274b007d29245882d22fa74248396ef377799978a Oct 29 23:29:12.425313 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.425275 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:29:12.425446 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.425368 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:29:12.425446 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.425416 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:29:12.522660 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.516911 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6"] Oct 29 23:29:12.535122 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.535061 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:29:12.535551 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.535451 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:29:12.535775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.535716 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:29:12.739093 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.739058 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf"] Oct 29 23:29:12.741146 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:29:12.741097 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6451f721_f9ac_4445_b719_bd6e21820d72.slice/crio-c84df660db39ecc09571002af3cd7c1fdd1f6cdabe95b6e3fc5a437f1a4ca1de WatchSource:0}: Error finding container c84df660db39ecc09571002af3cd7c1fdd1f6cdabe95b6e3fc5a437f1a4ca1de: Status 404 returned error can't find the container with id c84df660db39ecc09571002af3cd7c1fdd1f6cdabe95b6e3fc5a437f1a4ca1de Oct 29 23:29:12.745067 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.744807 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5"] Oct 29 23:29:12.746615 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.745794 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:29:12.746615 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.745878 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:29:12.746615 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.745929 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:29:12.750209 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.749764 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:29:12.750209 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.749829 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:29:12.750209 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:12.749869 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:29:13.177085 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.176978 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" event={"ID":"6451f721-f9ac-4445-b719-bd6e21820d72","Type":"ContainerStarted","Data":"efd22a131c4c954c080e1843e1f49b942f1233c71b0775d5ab24bb334534c242"} Oct 29 23:29:13.177085 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.177047 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" event={"ID":"6451f721-f9ac-4445-b719-bd6e21820d72","Type":"ContainerStarted","Data":"c84df660db39ecc09571002af3cd7c1fdd1f6cdabe95b6e3fc5a437f1a4ca1de"} Oct 29 23:29:13.179346 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.179300 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" event={"ID":"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4","Type":"ContainerStarted","Data":"af8b4d7dcd6f28d79776ff315ddc2e0678c21bb4d91d4cd2d4f6fcd34ab88227"} Oct 29 23:29:13.179548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.179360 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" event={"ID":"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4","Type":"ContainerStarted","Data":"8f55cc699504f6839fc6d42efbc30e0aebcad019487ad006d8e2a21e2ef1e080"} Oct 29 23:29:13.181724 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.181689 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" event={"ID":"039b4caf-d11b-4933-af15-2efb408c36e3","Type":"ContainerStarted","Data":"7633ea2be2aeb4c2d990f2afefdc49a55803208dded89996b6db4ae035bdb930"} Oct 29 23:29:13.181902 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.181735 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" event={"ID":"039b4caf-d11b-4933-af15-2efb408c36e3","Type":"ContainerStarted","Data":"6c93b60d4cf073aaa13cd4129ea17273a80dcd3b46b9fed258ae8be21831b58f"} Oct 29 23:29:13.183741 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.183709 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" event={"ID":"089367aa-c3e8-4f67-af00-697e5d7555bd","Type":"ContainerStarted","Data":"02c2a6d62d3edfe579b09c3bdc20737570e9c65ca27f5d270f09ec6c3f4bb0de"} Oct 29 23:29:13.183860 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.183770 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" event={"ID":"089367aa-c3e8-4f67-af00-697e5d7555bd","Type":"ContainerStarted","Data":"8b882730a80fc50a4bfa0ed274b007d29245882d22fa74248396ef377799978a"} Oct 29 23:29:13.217438 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.217062 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" podStartSLOduration=3.217044979 podStartE2EDuration="3.217044979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:13.214790259 +0000 UTC m=+922.577795006" watchObservedRunningTime="2026-10-29 23:29:13.217044979 +0000 UTC m=+922.580049699" Oct 29 23:29:13.252392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.252325 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" podStartSLOduration=3.252300973 podStartE2EDuration="3.252300973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:13.249630709 +0000 UTC m=+922.612635466" watchObservedRunningTime="2026-10-29 23:29:13.252300973 +0000 UTC m=+922.615305701" Oct 29 23:29:13.296231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.295687 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" podStartSLOduration=3.29564661 podStartE2EDuration="3.29564661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:13.290830415 +0000 UTC m=+922.653835140" watchObservedRunningTime="2026-10-29 23:29:13.29564661 +0000 UTC m=+922.658651350" Oct 29 23:29:13.350156 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.350068 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" podStartSLOduration=4.35004354 podStartE2EDuration="4.35004354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:13.327605309 +0000 UTC m=+922.690610052" watchObservedRunningTime="2026-10-29 23:29:13.35004354 +0000 UTC m=+922.713048264" Oct 29 23:29:13.403113 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.402967 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:13.408515 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.408086 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 29 23:29:13.408515 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.408164 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 29 23:29:13.433569 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.420506 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:13.433569 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.420664 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 23:29:13.433569 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:13.420722 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" podUID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 23:29:14.114481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:14.113986 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:14.115475 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:14.115152 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 29 23:29:14.115475 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:14.115210 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" podUID="089367aa-c3e8-4f67-af00-697e5d7555bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 29 23:29:14.184483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:14.184293 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:14.247630 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:14.202604 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 29 23:29:14.247630 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:14.202699 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" podUID="039b4caf-d11b-4933-af15-2efb408c36e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 29 23:29:14.470528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:14.460621 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 23:29:14.470528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:14.460718 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" podUID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 23:29:14.470528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:14.461292 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 29 23:29:14.470528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:14.461346 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 29 23:29:14.893718 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:14.893609 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 23:29:15.109485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:14.961205 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:15.109485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.063776 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:15.109485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.081607 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 29 23:29:15.109485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.107669 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" podUID="089367aa-c3e8-4f67-af00-697e5d7555bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 29 23:29:15.145541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.131806 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:15.146563 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.146523 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:29:15.297488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.286419 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:15.319306 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.317514 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 23:29:15.337496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.327983 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 29 23:29:15.337496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.328045 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" podUID="039b4caf-d11b-4933-af15-2efb408c36e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 29 23:29:15.456560 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.450092 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 23:29:15.456560 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.450680 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:15.456560 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.451007 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:15.496745 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.490707 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 29 23:29:15.496745 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.490786 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 29 23:29:15.496745 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.492229 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 23:29:15.496745 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.492272 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" podUID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 23:29:15.496745 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:15.493057 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:16.127811 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.127765 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 23:29:16.128269 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.128244 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 29 23:29:16.128585 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.128562 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:16.206530 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:29:16.206492 6507 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.415s" Oct 29 23:29:16.228481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.210667 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:29:16.228826 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.228801 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:29:16.228986 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.228975 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:29:16.229696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.229663 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:29:16.256490 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.231257 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:29:16.259240 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.259201 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 29 23:29:16.339485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.274578 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:29:16.339485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.329841 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:16.339485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.330239 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 23:29:16.339485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.330546 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:29:16.339485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.330633 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:29:16.339485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.334012 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 29 23:29:16.339485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.334056 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" podUID="089367aa-c3e8-4f67-af00-697e5d7555bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 29 23:29:16.365095 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.365039 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 29 23:29:16.375611 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.375563 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" podUID="039b4caf-d11b-4933-af15-2efb408c36e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 29 23:29:16.446575 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.421146 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 29 23:29:16.446575 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.421206 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 29 23:29:16.446575 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.429280 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 23:29:16.446575 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.429332 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" podUID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 23:29:16.473487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.459854 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:16.473487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.460056 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:29:16.473487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.460422 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 29 23:29:16.473487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.460874 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:29:16.473487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.465283 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 23:29:16.473487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.465652 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:29:16.493965 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.493922 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:16.499850 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:16.499817 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:29:16.734852 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:29:16.734749 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:29:17.146541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.145631 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:17.197479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.183400 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:17.213484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.207093 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:17.255483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.253649 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:17.281802 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.269352 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 29 23:29:17.298514 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.282079 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" podUID="089367aa-c3e8-4f67-af00-697e5d7555bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 29 23:29:17.298799 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.288664 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:17.330984 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.304917 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 29 23:29:17.330984 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.324598 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" podUID="039b4caf-d11b-4933-af15-2efb408c36e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 29 23:29:17.330984 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.325178 6507 trace.go:236] Trace[687517613]: "Calculate volume metrics of manifests for pod opendatahub/opendatahub-operator-controller-manager-bb7fb9d97-7bs76" (29-Oct-2026 23:29:16.299) (total time: 1025ms): Oct 29 23:29:17.330984 ip-10-0-133-100 kubenswrapper[6507]: Trace[687517613]: [1.025867318s] [1.025867318s] END Oct 29 23:29:17.331546 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.331510 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:17.342872 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.342834 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 23:29:17.430518 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.430408 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 29 23:29:17.430896 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.430855 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 29 23:29:17.491484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.491432 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 23:29:17.491780 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.491733 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" podUID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 23:29:17.506621 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.506559 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 29 23:29:17.527531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.510579 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:17.527531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.520082 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 23:29:17.577482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.539663 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 29 23:29:17.812484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.782670 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded" Oct 29 23:29:17.907472 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:17.907423 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:29:18.092478 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.092360 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 29 23:29:18.100283 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.100227 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" podUID="089367aa-c3e8-4f67-af00-697e5d7555bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 29 23:29:18.178592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.178552 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 29 23:29:18.181567 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.181527 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" podUID="039b4caf-d11b-4933-af15-2efb408c36e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 29 23:29:18.209555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.209513 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:18.210106 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.210076 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 29 23:29:18.229827 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.229789 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:18.275164 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.272583 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:29:18.312981 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.312879 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 29 23:29:18.346937 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.338230 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:18.381612 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.381560 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:18.478421 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.469506 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 23:29:18.478421 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.469570 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" podUID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 23:29:18.478421 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.469663 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 29 23:29:18.478421 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.469684 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 29 23:29:18.518123 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.518081 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:18.521637 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.521599 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:18.542853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.542810 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:18.543162 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.542810 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:18.581747 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.581705 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:18.658537 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:18.658258 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:19.158596 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.158558 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 29 23:29:19.158903 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.158871 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" podUID="089367aa-c3e8-4f67-af00-697e5d7555bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 29 23:29:19.189852 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.189815 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 29 23:29:19.190121 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.190093 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" podUID="039b4caf-d11b-4933-af15-2efb408c36e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 29 23:29:19.214234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.214188 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:19.215767 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.215539 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 23:29:19.283656 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.283599 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:19.286086 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.286049 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:29:19.366541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.350547 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:19.368098 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.368059 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 23:29:19.396525 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.390529 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:19.402113 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.402078 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 29 23:29:19.406846 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.406808 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 29 23:29:19.419567 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.419483 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 23:29:19.419815 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.419786 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" podUID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 23:29:19.527828 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.525971 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:19.539629 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.528160 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:19.539629 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.528451 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 29 23:29:19.629483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.604054 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:19.801501 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:19.744834 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:29:20.390743 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.390679 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:20.533485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.391341 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Oct 29 23:29:20.533485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.464873 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 23:29:20.533485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.470951 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:20.533485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.489634 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 29 23:29:20.533485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.491160 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" podUID="039b4caf-d11b-4933-af15-2efb408c36e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 29 23:29:20.533485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.490224 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 29 23:29:20.533485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.491227 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" podUID="089367aa-c3e8-4f67-af00-697e5d7555bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 29 23:29:20.533485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.493329 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 29 23:29:20.533485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.493385 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 29 23:29:20.534245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.493450 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:20.657417 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.657308 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 29 23:29:20.661550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.661479 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:20.662372 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.662348 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:29:20.759483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.716681 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:20.873480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.868624 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 23:29:20.873480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.868713 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" podUID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 23:29:20.873480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.869739 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:20.873480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.869820 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 29 23:29:20.873480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:20.870305 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:29:21.129947 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.129903 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z7xww" podUID="d8489303-190c-4dcf-8deb-125ecb21210c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded" Oct 29 23:29:21.249482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.248835 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:29:21.270837 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.270707 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:29:21.271145 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.271126 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:29:21.284573 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.284546 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 29 23:29:21.284799 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.284775 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" podUID="089367aa-c3e8-4f67-af00-697e5d7555bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 29 23:29:21.297477 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.287816 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:29:21.372556 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.370926 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 29 23:29:21.372556 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.370988 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" podUID="039b4caf-d11b-4933-af15-2efb408c36e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 29 23:29:21.412599 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.407484 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 29 23:29:21.412599 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.407678 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 29 23:29:21.418341 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.418304 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 23:29:21.418647 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.418614 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" podUID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 23:29:21.471106 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.471072 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:29:21.524679 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.524645 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:29:21.587663 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.587606 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:21.663669 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.600261 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:29:21.663669 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.660928 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:29:21.663669 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.662015 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:29:21.690616 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:21.690388 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:29:22.089039 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:22.087025 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 29 23:29:22.089039 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:22.087087 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" podUID="089367aa-c3e8-4f67-af00-697e5d7555bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 29 23:29:22.186104 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:22.175735 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 29 23:29:22.186104 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:22.175794 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" podUID="039b4caf-d11b-4933-af15-2efb408c36e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 29 23:29:22.403537 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:22.402875 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 29 23:29:22.403537 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:22.402954 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 29 23:29:22.424482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:22.413334 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 23:29:22.424482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:22.413400 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" podUID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 23:29:23.083483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.079311 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7"] Oct 29 23:29:23.089513 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.088893 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7" Oct 29 23:29:23.107433 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.106561 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:29:23.118773 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.117623 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:29:23.118773 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.117745 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:29:23.118773 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.117805 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:29:23.118773 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.117811 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:23.119509 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.119283 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:23.127948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.127915 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:29:23.144487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.130136 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efed76ac-e277-4ad2-bf5d-7924ad9bd181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-935b71b4-zthz7\" (UID: \"efed76ac-e277-4ad2-bf5d-7924ad9bd181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7" Oct 29 23:29:23.144487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.130351 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h87h5\" (UniqueName: \"kubernetes.io/projected/efed76ac-e277-4ad2-bf5d-7924ad9bd181-kube-api-access-h87h5\") pod \"maas-api-revoke-sub-935b71b4-zthz7\" (UID: \"efed76ac-e277-4ad2-bf5d-7924ad9bd181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7" Oct 29 23:29:23.144487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.130437 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efed76ac-e277-4ad2-bf5d-7924ad9bd181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-935b71b4-zthz7\" (UID: \"efed76ac-e277-4ad2-bf5d-7924ad9bd181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7" Oct 29 23:29:23.144487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.136051 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7"] Oct 29 23:29:23.315483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.311349 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efed76ac-e277-4ad2-bf5d-7924ad9bd181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-935b71b4-zthz7\" (UID: \"efed76ac-e277-4ad2-bf5d-7924ad9bd181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7" Oct 29 23:29:23.348785 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.321094 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efed76ac-e277-4ad2-bf5d-7924ad9bd181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-935b71b4-zthz7\" (UID: \"efed76ac-e277-4ad2-bf5d-7924ad9bd181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7" Oct 29 23:29:23.348785 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.336705 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h87h5\" (UniqueName: \"kubernetes.io/projected/efed76ac-e277-4ad2-bf5d-7924ad9bd181-kube-api-access-h87h5\") pod \"maas-api-revoke-sub-935b71b4-zthz7\" (UID: \"efed76ac-e277-4ad2-bf5d-7924ad9bd181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7" Oct 29 23:29:23.348785 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.336802 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efed76ac-e277-4ad2-bf5d-7924ad9bd181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-935b71b4-zthz7\" (UID: \"efed76ac-e277-4ad2-bf5d-7924ad9bd181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7" Oct 29 23:29:23.365156 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.365119 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:23.442763 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.412332 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:23.442763 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.433865 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:29:23.442763 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.438603 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h87h5\" (UniqueName: \"kubernetes.io/projected/efed76ac-e277-4ad2-bf5d-7924ad9bd181-kube-api-access-h87h5\") pod \"maas-api-revoke-sub-935b71b4-zthz7\" (UID: \"efed76ac-e277-4ad2-bf5d-7924ad9bd181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7" Oct 29 23:29:23.442763 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.440565 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efed76ac-e277-4ad2-bf5d-7924ad9bd181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-935b71b4-zthz7\" (UID: \"efed76ac-e277-4ad2-bf5d-7924ad9bd181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7" Oct 29 23:29:23.483994 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.481352 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:23.483994 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.482826 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:23.485228 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.485203 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:29:23.729831 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.729792 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7" Oct 29 23:29:23.834720 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.833738 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:23.834720 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.834530 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:23.835297 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:23.835275 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:29:24.043978 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:29:24.043939 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefed76ac_e277_4ad2_bf5d_7924ad9bd181.slice/crio-e03906868727856f269632e890fc95349df6576a509aa78ce47d74d6e8295e5d WatchSource:0}: Error finding container e03906868727856f269632e890fc95349df6576a509aa78ce47d74d6e8295e5d: Status 404 returned error can't find the container with id e03906868727856f269632e890fc95349df6576a509aa78ce47d74d6e8295e5d Oct 29 23:29:24.044593 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:24.044569 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7"] Oct 29 23:29:25.002326 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:25.002293 6507 generic.go:356] "Generic (PLEG): container finished" podID="efed76ac-e277-4ad2-bf5d-7924ad9bd181" containerID="fef7f17b0b6fa8017c379daa3268b488f0a8754d31eccd67d702b720a04290d9" exitCode=0 Oct 29 23:29:25.002816 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:25.002351 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7" event={"ID":"efed76ac-e277-4ad2-bf5d-7924ad9bd181","Type":"ContainerDied","Data":"fef7f17b0b6fa8017c379daa3268b488f0a8754d31eccd67d702b720a04290d9"} Oct 29 23:29:25.002816 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:25.002391 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7" event={"ID":"efed76ac-e277-4ad2-bf5d-7924ad9bd181","Type":"ContainerStarted","Data":"e03906868727856f269632e890fc95349df6576a509aa78ce47d74d6e8295e5d"} Oct 29 23:29:26.184527 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.184499 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7" Oct 29 23:29:26.311985 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.311101 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efed76ac-e277-4ad2-bf5d-7924ad9bd181-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efed76ac-e277-4ad2-bf5d-7924ad9bd181-maas-api-openshift-ca\") pod \"efed76ac-e277-4ad2-bf5d-7924ad9bd181\" (UID: \"efed76ac-e277-4ad2-bf5d-7924ad9bd181\") " Oct 29 23:29:26.311985 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.311267 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efed76ac-e277-4ad2-bf5d-7924ad9bd181-kube-api-access-h87h5\" (UniqueName: \"kubernetes.io/projected/efed76ac-e277-4ad2-bf5d-7924ad9bd181-kube-api-access-h87h5\") pod \"efed76ac-e277-4ad2-bf5d-7924ad9bd181\" (UID: \"efed76ac-e277-4ad2-bf5d-7924ad9bd181\") " Oct 29 23:29:26.311985 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.311345 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efed76ac-e277-4ad2-bf5d-7924ad9bd181-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efed76ac-e277-4ad2-bf5d-7924ad9bd181-maas-api-xks-ca\") pod \"efed76ac-e277-4ad2-bf5d-7924ad9bd181\" (UID: \"efed76ac-e277-4ad2-bf5d-7924ad9bd181\") " Oct 29 23:29:26.328318 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.313845 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efed76ac-e277-4ad2-bf5d-7924ad9bd181-maas-api-openshift-ca" pod "efed76ac-e277-4ad2-bf5d-7924ad9bd181" (UID: "efed76ac-e277-4ad2-bf5d-7924ad9bd181"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:26.348249 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.348205 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efed76ac-e277-4ad2-bf5d-7924ad9bd181-maas-api-xks-ca" pod "efed76ac-e277-4ad2-bf5d-7924ad9bd181" (UID: "efed76ac-e277-4ad2-bf5d-7924ad9bd181"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:26.348437 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.348274 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7"] Oct 29 23:29:26.349373 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.349293 6507 cpu_manager.go:414] "containerMap: removing container" podUID="efed76ac-e277-4ad2-bf5d-7924ad9bd181" containerName="revoke-keys" Oct 29 23:29:26.395498 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.394569 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efed76ac-e277-4ad2-bf5d-7924ad9bd181" containerName="revoke-keys" Oct 29 23:29:26.395498 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.394949 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="efed76ac-e277-4ad2-bf5d-7924ad9bd181" containerName="revoke-keys" Oct 29 23:29:26.406793 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.382800 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efed76ac-e277-4ad2-bf5d-7924ad9bd181-kube-api-access-h87h5" pod "efed76ac-e277-4ad2-bf5d-7924ad9bd181" (UID: "efed76ac-e277-4ad2-bf5d-7924ad9bd181"). InnerVolumeSpecName "kube-api-access-h87h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:26.410124 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.410095 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7" Oct 29 23:29:26.412177 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.412147 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7"] Oct 29 23:29:26.413808 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.413769 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h87h5\" (UniqueName: \"kubernetes.io/projected/efed76ac-e277-4ad2-bf5d-7924ad9bd181-kube-api-access-h87h5\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:26.413936 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.413811 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efed76ac-e277-4ad2-bf5d-7924ad9bd181-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:26.413936 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.413839 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efed76ac-e277-4ad2-bf5d-7924ad9bd181-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:26.514959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.514923 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ssv\" (UniqueName: \"kubernetes.io/projected/28a10cdb-a7bc-423f-97f3-deb0574c3540-kube-api-access-f9ssv\") pod \"maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7\" (UID: \"28a10cdb-a7bc-423f-97f3-deb0574c3540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7" Oct 29 23:29:26.515137 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.515058 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a10cdb-a7bc-423f-97f3-deb0574c3540-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7\" (UID: \"28a10cdb-a7bc-423f-97f3-deb0574c3540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7" Oct 29 23:29:26.515137 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.515119 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a10cdb-a7bc-423f-97f3-deb0574c3540-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7\" (UID: \"28a10cdb-a7bc-423f-97f3-deb0574c3540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7" Oct 29 23:29:26.617032 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.616937 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ssv\" (UniqueName: \"kubernetes.io/projected/28a10cdb-a7bc-423f-97f3-deb0574c3540-kube-api-access-f9ssv\") pod \"maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7\" (UID: \"28a10cdb-a7bc-423f-97f3-deb0574c3540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7" Oct 29 23:29:26.617194 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.617040 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a10cdb-a7bc-423f-97f3-deb0574c3540-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7\" (UID: \"28a10cdb-a7bc-423f-97f3-deb0574c3540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7" Oct 29 23:29:26.617194 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.617059 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a10cdb-a7bc-423f-97f3-deb0574c3540-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7\" (UID: \"28a10cdb-a7bc-423f-97f3-deb0574c3540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7" Oct 29 23:29:26.617787 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.617764 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a10cdb-a7bc-423f-97f3-deb0574c3540-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7\" (UID: \"28a10cdb-a7bc-423f-97f3-deb0574c3540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7" Oct 29 23:29:26.619492 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.619444 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a10cdb-a7bc-423f-97f3-deb0574c3540-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7\" (UID: \"28a10cdb-a7bc-423f-97f3-deb0574c3540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7" Oct 29 23:29:26.633594 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.633571 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ssv\" (UniqueName: \"kubernetes.io/projected/28a10cdb-a7bc-423f-97f3-deb0574c3540-kube-api-access-f9ssv\") pod \"maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7\" (UID: \"28a10cdb-a7bc-423f-97f3-deb0574c3540\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7" Oct 29 23:29:26.747368 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:26.747334 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7" Oct 29 23:29:27.018881 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:27.018855 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7" Oct 29 23:29:27.019047 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:27.018897 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7" event={"ID":"efed76ac-e277-4ad2-bf5d-7924ad9bd181","Type":"ContainerDied","Data":"e03906868727856f269632e890fc95349df6576a509aa78ce47d74d6e8295e5d"} Oct 29 23:29:27.019047 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:27.018949 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e03906868727856f269632e890fc95349df6576a509aa78ce47d74d6e8295e5d" Oct 29 23:29:27.059320 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:27.059288 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7"] Oct 29 23:29:27.060918 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:29:27.060882 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a10cdb_a7bc_423f_97f3_deb0574c3540.slice/crio-9b7fcf6afda667f73611208c00d7cbddb41ae32c114f2f9a0dde8528e3ecb818 WatchSource:0}: Error finding container 9b7fcf6afda667f73611208c00d7cbddb41ae32c114f2f9a0dde8528e3ecb818: Status 404 returned error can't find the container with id 9b7fcf6afda667f73611208c00d7cbddb41ae32c114f2f9a0dde8528e3ecb818 Oct 29 23:29:28.029424 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:28.029390 6507 generic.go:356] "Generic (PLEG): container finished" podID="28a10cdb-a7bc-423f-97f3-deb0574c3540" containerID="2da4af94c242845006097def6d19755b859e6c94ec51fdec7bcbc3c10fcb0135" exitCode=0 Oct 29 23:29:28.029840 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:28.029443 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7" event={"ID":"28a10cdb-a7bc-423f-97f3-deb0574c3540","Type":"ContainerDied","Data":"2da4af94c242845006097def6d19755b859e6c94ec51fdec7bcbc3c10fcb0135"} Oct 29 23:29:28.029840 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:28.029510 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7" event={"ID":"28a10cdb-a7bc-423f-97f3-deb0574c3540","Type":"ContainerStarted","Data":"9b7fcf6afda667f73611208c00d7cbddb41ae32c114f2f9a0dde8528e3ecb818"} Oct 29 23:29:29.218143 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:29.218113 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7" Oct 29 23:29:29.258004 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:29.257971 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a10cdb-a7bc-423f-97f3-deb0574c3540-kube-api-access-f9ssv\" (UniqueName: \"kubernetes.io/projected/28a10cdb-a7bc-423f-97f3-deb0574c3540-kube-api-access-f9ssv\") pod \"28a10cdb-a7bc-423f-97f3-deb0574c3540\" (UID: \"28a10cdb-a7bc-423f-97f3-deb0574c3540\") " Oct 29 23:29:29.258205 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:29.258159 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a10cdb-a7bc-423f-97f3-deb0574c3540-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a10cdb-a7bc-423f-97f3-deb0574c3540-maas-api-openshift-ca\") pod \"28a10cdb-a7bc-423f-97f3-deb0574c3540\" (UID: \"28a10cdb-a7bc-423f-97f3-deb0574c3540\") " Oct 29 23:29:29.258276 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:29.258203 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28a10cdb-a7bc-423f-97f3-deb0574c3540-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a10cdb-a7bc-423f-97f3-deb0574c3540-maas-api-xks-ca\") pod \"28a10cdb-a7bc-423f-97f3-deb0574c3540\" (UID: \"28a10cdb-a7bc-423f-97f3-deb0574c3540\") " Oct 29 23:29:29.258614 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:29.258587 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a10cdb-a7bc-423f-97f3-deb0574c3540-maas-api-openshift-ca" pod "28a10cdb-a7bc-423f-97f3-deb0574c3540" (UID: "28a10cdb-a7bc-423f-97f3-deb0574c3540"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:29.258928 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:29.258908 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a10cdb-a7bc-423f-97f3-deb0574c3540-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:29.260212 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:29.260180 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a10cdb-a7bc-423f-97f3-deb0574c3540-kube-api-access-f9ssv" pod "28a10cdb-a7bc-423f-97f3-deb0574c3540" (UID: "28a10cdb-a7bc-423f-97f3-deb0574c3540"). InnerVolumeSpecName "kube-api-access-f9ssv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:29.260346 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:29.260320 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a10cdb-a7bc-423f-97f3-deb0574c3540-maas-api-xks-ca" pod "28a10cdb-a7bc-423f-97f3-deb0574c3540" (UID: "28a10cdb-a7bc-423f-97f3-deb0574c3540"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:29.359970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:29.359882 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a10cdb-a7bc-423f-97f3-deb0574c3540-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:29.359970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:29.359914 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9ssv\" (UniqueName: \"kubernetes.io/projected/28a10cdb-a7bc-423f-97f3-deb0574c3540-kube-api-access-f9ssv\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:30.049007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:30.048974 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7" Oct 29 23:29:30.049007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:30.049001 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7" event={"ID":"28a10cdb-a7bc-423f-97f3-deb0574c3540","Type":"ContainerDied","Data":"9b7fcf6afda667f73611208c00d7cbddb41ae32c114f2f9a0dde8528e3ecb818"} Oct 29 23:29:30.049261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:30.049050 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b7fcf6afda667f73611208c00d7cbddb41ae32c114f2f9a0dde8528e3ecb818" Oct 29 23:29:31.173479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:31.157006 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:29:31.175545 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:31.173881 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx"] Oct 29 23:29:31.200180 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:31.200150 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc"] Oct 29 23:29:33.319630 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:33.319574 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 29 23:29:33.341737 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:33.330582 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:33.342170 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:33.342131 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:33.401518 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:33.397149 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:34.037484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.037385 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:34.164504 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.162568 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:34.249487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.234613 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:34.249487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.235063 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 29 23:29:34.273066 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.273000 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:34.290598 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.290452 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:34.341616 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.341560 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 23:29:34.343074 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.343044 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:29:34.351520 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.343734 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:34.373881 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.343773 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:34.375204 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.374143 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:29:34.375921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.375900 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:29:34.404653 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.398129 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:34.404653 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.398644 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:29:34.529569 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.520932 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:34.545609 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.535673 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:34.567521 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.558519 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:34.613699 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.607974 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:34.873556 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.870561 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:29:34.873556 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:34.870628 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" podUID="d02ada27-df84-4ea9-988d-3e3f0516f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:35.174542 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.173078 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 29 23:29:35.208031 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.194945 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:29:35.378491 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.376583 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:35.395517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.379098 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 29 23:29:35.502596 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.400543 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:35.502596 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.400627 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:35.502596 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.400707 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:35.502596 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.400715 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:35.502596 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.401064 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:35.502596 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.401437 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:29:35.502596 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.401524 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:29:35.502596 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.401564 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:29:35.654480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.633783 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 29 23:29:35.716481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.716426 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:29:35.803537 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.781995 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 29 23:29:36.020086 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.771237 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 29 23:29:36.021453 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:36.021428 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:29:36.021723 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:36.021708 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:29:36.054626 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:35.923303 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:29:36.095483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:36.091581 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" podUID="914b9029-3376-4fad-addf-ce1b73391b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:36.190484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:36.186582 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:36.404907 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:36.404806 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:36.704486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:36.405445 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:36.704486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:36.407098 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:36.704486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:36.407134 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:36.704486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:36.407166 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 23:29:36.704486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:36.407186 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:36.704486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:36.407206 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:36.704486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:36.663719 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:37.143595 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:37.055790 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 29 23:29:37.170760 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:37.056610 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 29 23:29:37.282484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:37.257813 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 29 23:29:37.721031 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:37.712582 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:37.721031 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:37.715600 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:37.721031 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:37.715955 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:37.721031 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:37.716254 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:37.733491 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:37.721937 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 29 23:29:37.733491 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:37.725560 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:37.785757 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:37.740543 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:37.785757 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:37.741756 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:37.882115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:37.861883 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 23:29:38.204993 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:38.173004 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:38.204993 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:38.173371 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 29 23:29:38.304645 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:38.304600 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 29 23:29:38.359310 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:38.359278 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:29:38.747401 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:38.733108 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:38.747401 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:38.733529 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 29 23:29:38.747401 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:38.738113 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:38.747401 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:38.738878 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 29 23:29:38.747401 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:38.741614 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:38.772685 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:38.759348 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:38.772685 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:38.759686 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 29 23:29:38.772685 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:38.772069 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:38.881484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:38.869556 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:38.982484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:38.903910 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:39.212493 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:39.211618 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:29:39.481559 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:39.477591 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" podUID="089367aa-c3e8-4f67-af00-697e5d7555bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:39.481559 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:39.219627 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 29 23:29:39.481559 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:39.215634 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:39.481559 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:39.457551 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:39.481559 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:39.457607 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 29 23:29:39.611489 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:39.598946 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:29:39.611489 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:39.599728 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:39.734554 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:39.733584 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 23:29:39.734554 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:39.733741 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 29 23:29:39.756450 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:39.756407 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:39.775618 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:39.775569 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:39.777423 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:39.757032 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 23:29:39.911636 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:39.911585 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:39.972759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:39.972727 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:29:39.990716 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:39.990615 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:29:40.000572 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.000538 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc"] Oct 29 23:29:40.047537 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.047507 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:29:40.153477 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.152209 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:29:40.159512 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.159333 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:29:40.162123 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.162101 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:29:40.167150 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.166534 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7"] Oct 29 23:29:40.167150 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.167087 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:29:40.169221 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.167587 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:29:40.180254 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.180221 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:29:40.188448 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.187712 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:29:40.252942 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.252864 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:29:40.326042 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.325997 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc"] Oct 29 23:29:40.327062 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.327033 6507 cpu_manager.go:414] "containerMap: removing container" podUID="28a10cdb-a7bc-423f-97f3-deb0574c3540" containerName="revoke-keys" Oct 29 23:29:40.327062 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.327058 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a10cdb-a7bc-423f-97f3-deb0574c3540" containerName="revoke-keys" Oct 29 23:29:40.327348 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.327331 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a10cdb-a7bc-423f-97f3-deb0574c3540" containerName="revoke-keys" Oct 29 23:29:40.332560 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.332530 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc" Oct 29 23:29:40.353482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.352737 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:29:40.353482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.352891 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:29:40.353482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.352941 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:29:40.353482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.352997 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:29:40.356481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.356183 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc"] Oct 29 23:29:40.456266 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.456236 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69bec15-7520-4331-a8af-bf1cefd3fd1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01e07b87-c54jc\" (UID: \"e69bec15-7520-4331-a8af-bf1cefd3fd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc" Oct 29 23:29:40.456445 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.456285 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6d29\" (UniqueName: \"kubernetes.io/projected/e69bec15-7520-4331-a8af-bf1cefd3fd1e-kube-api-access-k6d29\") pod \"maas-api-revoke-sub-01e07b87-c54jc\" (UID: \"e69bec15-7520-4331-a8af-bf1cefd3fd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc" Oct 29 23:29:40.456562 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.456527 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69bec15-7520-4331-a8af-bf1cefd3fd1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01e07b87-c54jc\" (UID: \"e69bec15-7520-4331-a8af-bf1cefd3fd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc" Oct 29 23:29:40.558691 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.558595 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69bec15-7520-4331-a8af-bf1cefd3fd1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01e07b87-c54jc\" (UID: \"e69bec15-7520-4331-a8af-bf1cefd3fd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc" Oct 29 23:29:40.558691 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.558652 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6d29\" (UniqueName: \"kubernetes.io/projected/e69bec15-7520-4331-a8af-bf1cefd3fd1e-kube-api-access-k6d29\") pod \"maas-api-revoke-sub-01e07b87-c54jc\" (UID: \"e69bec15-7520-4331-a8af-bf1cefd3fd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc" Oct 29 23:29:40.558900 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.558809 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69bec15-7520-4331-a8af-bf1cefd3fd1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01e07b87-c54jc\" (UID: \"e69bec15-7520-4331-a8af-bf1cefd3fd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc" Oct 29 23:29:40.559528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.559505 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69bec15-7520-4331-a8af-bf1cefd3fd1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01e07b87-c54jc\" (UID: \"e69bec15-7520-4331-a8af-bf1cefd3fd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc" Oct 29 23:29:40.561338 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.561313 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69bec15-7520-4331-a8af-bf1cefd3fd1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01e07b87-c54jc\" (UID: \"e69bec15-7520-4331-a8af-bf1cefd3fd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc" Oct 29 23:29:40.578968 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.578931 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6d29\" (UniqueName: \"kubernetes.io/projected/e69bec15-7520-4331-a8af-bf1cefd3fd1e-kube-api-access-k6d29\") pod \"maas-api-revoke-sub-01e07b87-c54jc\" (UID: \"e69bec15-7520-4331-a8af-bf1cefd3fd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc" Oct 29 23:29:40.649759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.649723 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc" Oct 29 23:29:40.973099 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:40.973072 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc"] Oct 29 23:29:40.975317 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:29:40.975288 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69bec15_7520_4331_a8af_bf1cefd3fd1e.slice/crio-916103c1027ec585c06e9f05ffd901b03377f059b668ce6f4ac380a4bf4e1169 WatchSource:0}: Error finding container 916103c1027ec585c06e9f05ffd901b03377f059b668ce6f4ac380a4bf4e1169: Status 404 returned error can't find the container with id 916103c1027ec585c06e9f05ffd901b03377f059b668ce6f4ac380a4bf4e1169 Oct 29 23:29:41.161409 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:41.161376 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc" event={"ID":"e69bec15-7520-4331-a8af-bf1cefd3fd1e","Type":"ContainerStarted","Data":"916103c1027ec585c06e9f05ffd901b03377f059b668ce6f4ac380a4bf4e1169"} Oct 29 23:29:42.171678 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:42.171636 6507 generic.go:356] "Generic (PLEG): container finished" podID="e69bec15-7520-4331-a8af-bf1cefd3fd1e" containerID="93a5ded4e82bc6e030b35a0c6d398956d4db5b5eeba214c48a82f482c8ad7171" exitCode=0 Oct 29 23:29:42.172174 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:42.171682 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc" event={"ID":"e69bec15-7520-4331-a8af-bf1cefd3fd1e","Type":"ContainerDied","Data":"93a5ded4e82bc6e030b35a0c6d398956d4db5b5eeba214c48a82f482c8ad7171"} Oct 29 23:29:42.514811 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:42.514774 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6"] Oct 29 23:29:42.519231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:42.519213 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6" Oct 29 23:29:42.540539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:42.540502 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6"] Oct 29 23:29:42.610967 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:42.610934 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e99cfdf5-lg6l6\" (UID: \"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6" Oct 29 23:29:42.611152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:42.611086 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74svk\" (UniqueName: \"kubernetes.io/projected/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-kube-api-access-74svk\") pod \"maas-api-revoke-sub-e99cfdf5-lg6l6\" (UID: \"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6" Oct 29 23:29:42.611152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:42.611149 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e99cfdf5-lg6l6\" (UID: \"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6" Oct 29 23:29:42.712098 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:42.712065 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e99cfdf5-lg6l6\" (UID: \"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6" Oct 29 23:29:42.712301 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:42.712196 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74svk\" (UniqueName: \"kubernetes.io/projected/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-kube-api-access-74svk\") pod \"maas-api-revoke-sub-e99cfdf5-lg6l6\" (UID: \"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6" Oct 29 23:29:42.712301 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:42.712256 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e99cfdf5-lg6l6\" (UID: \"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6" Oct 29 23:29:42.712891 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:42.712864 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e99cfdf5-lg6l6\" (UID: \"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6" Oct 29 23:29:42.714804 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:42.714784 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e99cfdf5-lg6l6\" (UID: \"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6" Oct 29 23:29:42.729229 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:42.729196 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74svk\" (UniqueName: \"kubernetes.io/projected/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-kube-api-access-74svk\") pod \"maas-api-revoke-sub-e99cfdf5-lg6l6\" (UID: \"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6" Oct 29 23:29:42.833164 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:42.833066 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6" Oct 29 23:29:43.241242 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:43.240949 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6"] Oct 29 23:29:43.252515 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:29:43.242432 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60bf6b38_c0c8_4e8b_bd72_6876edf4dae3.slice/crio-f37e1716851c1a607451fc4169b81f7e0e9c3f091090d500e4a2e28570bc9096 WatchSource:0}: Error finding container f37e1716851c1a607451fc4169b81f7e0e9c3f091090d500e4a2e28570bc9096: Status 404 returned error can't find the container with id f37e1716851c1a607451fc4169b81f7e0e9c3f091090d500e4a2e28570bc9096 Oct 29 23:29:43.326081 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:43.326052 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc" Oct 29 23:29:43.432634 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:43.432554 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69bec15-7520-4331-a8af-bf1cefd3fd1e-kube-api-access-k6d29\" (UniqueName: \"kubernetes.io/projected/e69bec15-7520-4331-a8af-bf1cefd3fd1e-kube-api-access-k6d29\") pod \"e69bec15-7520-4331-a8af-bf1cefd3fd1e\" (UID: \"e69bec15-7520-4331-a8af-bf1cefd3fd1e\") " Oct 29 23:29:43.432785 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:43.432636 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e69bec15-7520-4331-a8af-bf1cefd3fd1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69bec15-7520-4331-a8af-bf1cefd3fd1e-maas-api-xks-ca\") pod \"e69bec15-7520-4331-a8af-bf1cefd3fd1e\" (UID: \"e69bec15-7520-4331-a8af-bf1cefd3fd1e\") " Oct 29 23:29:43.432785 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:43.432675 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e69bec15-7520-4331-a8af-bf1cefd3fd1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69bec15-7520-4331-a8af-bf1cefd3fd1e-maas-api-openshift-ca\") pod \"e69bec15-7520-4331-a8af-bf1cefd3fd1e\" (UID: \"e69bec15-7520-4331-a8af-bf1cefd3fd1e\") " Oct 29 23:29:43.433071 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:43.433048 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69bec15-7520-4331-a8af-bf1cefd3fd1e-maas-api-openshift-ca" pod "e69bec15-7520-4331-a8af-bf1cefd3fd1e" (UID: "e69bec15-7520-4331-a8af-bf1cefd3fd1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:43.433427 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:43.433407 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e69bec15-7520-4331-a8af-bf1cefd3fd1e-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:43.434694 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:43.434663 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69bec15-7520-4331-a8af-bf1cefd3fd1e-maas-api-xks-ca" pod "e69bec15-7520-4331-a8af-bf1cefd3fd1e" (UID: "e69bec15-7520-4331-a8af-bf1cefd3fd1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:43.434762 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:43.434739 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69bec15-7520-4331-a8af-bf1cefd3fd1e-kube-api-access-k6d29" pod "e69bec15-7520-4331-a8af-bf1cefd3fd1e" (UID: "e69bec15-7520-4331-a8af-bf1cefd3fd1e"). InnerVolumeSpecName "kube-api-access-k6d29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:43.534882 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:43.534848 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6d29\" (UniqueName: \"kubernetes.io/projected/e69bec15-7520-4331-a8af-bf1cefd3fd1e-kube-api-access-k6d29\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:43.534882 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:43.534879 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e69bec15-7520-4331-a8af-bf1cefd3fd1e-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:44.190191 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:44.190153 6507 generic.go:356] "Generic (PLEG): container finished" podID="60bf6b38-c0c8-4e8b-bd72-6876edf4dae3" containerID="161fc1f1be53862ceb08a106094348c1d0f2bac83b48164cbbe6f3631450604c" exitCode=0 Oct 29 23:29:44.190388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:44.190235 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6" event={"ID":"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3","Type":"ContainerDied","Data":"161fc1f1be53862ceb08a106094348c1d0f2bac83b48164cbbe6f3631450604c"} Oct 29 23:29:44.190388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:44.190288 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6" event={"ID":"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3","Type":"ContainerStarted","Data":"f37e1716851c1a607451fc4169b81f7e0e9c3f091090d500e4a2e28570bc9096"} Oct 29 23:29:44.191801 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:44.191775 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc" event={"ID":"e69bec15-7520-4331-a8af-bf1cefd3fd1e","Type":"ContainerDied","Data":"916103c1027ec585c06e9f05ffd901b03377f059b668ce6f4ac380a4bf4e1169"} Oct 29 23:29:44.191932 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:44.191800 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc" Oct 29 23:29:44.191932 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:44.191821 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="916103c1027ec585c06e9f05ffd901b03377f059b668ce6f4ac380a4bf4e1169" Oct 29 23:29:45.001562 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:45.001515 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" podUID="6173dedb-87b4-4028-9506-34ffadb93263" containerName="payload-processing" containerID="cri-o://50615357a1b5beac4da23d76a90f5e586e298be3251d9682c006396ef94685ec" gracePeriod=25 Oct 29 23:29:45.167353 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:45.167298 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" podUID="9c730892-1170-44b8-9fcf-e74a43035722" containerName="payload-pre-processing" containerID="cri-o://d3b14bd874fd59a3a847a785864271dc7273082d9b3253319e41e78f2cbb0f0d" gracePeriod=25 Oct 29 23:29:45.227536 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:29:45.227497 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:29:45.358372 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:45.358339 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6" Oct 29 23:29:45.521609 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:45.521523 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-kube-api-access-74svk\" (UniqueName: \"kubernetes.io/projected/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-kube-api-access-74svk\") pod \"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3\" (UID: \"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3\") " Oct 29 23:29:45.521761 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:45.521690 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-maas-api-openshift-ca\") pod \"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3\" (UID: \"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3\") " Oct 29 23:29:45.521802 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:45.521755 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-maas-api-xks-ca\") pod \"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3\" (UID: \"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3\") " Oct 29 23:29:45.522137 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:45.522090 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-maas-api-openshift-ca" pod "60bf6b38-c0c8-4e8b-bd72-6876edf4dae3" (UID: "60bf6b38-c0c8-4e8b-bd72-6876edf4dae3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:45.522728 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:45.522700 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:45.523923 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:45.523896 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-kube-api-access-74svk" pod "60bf6b38-c0c8-4e8b-bd72-6876edf4dae3" (UID: "60bf6b38-c0c8-4e8b-bd72-6876edf4dae3"). InnerVolumeSpecName "kube-api-access-74svk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:45.524004 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:45.523971 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-maas-api-xks-ca" pod "60bf6b38-c0c8-4e8b-bd72-6876edf4dae3" (UID: "60bf6b38-c0c8-4e8b-bd72-6876edf4dae3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:45.624536 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:45.624502 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74svk\" (UniqueName: \"kubernetes.io/projected/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-kube-api-access-74svk\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:45.624536 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:45.624535 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:46.209945 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.209908 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6" event={"ID":"60bf6b38-c0c8-4e8b-bd72-6876edf4dae3","Type":"ContainerDied","Data":"f37e1716851c1a607451fc4169b81f7e0e9c3f091090d500e4a2e28570bc9096"} Oct 29 23:29:46.210588 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.210000 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f37e1716851c1a607451fc4169b81f7e0e9c3f091090d500e4a2e28570bc9096" Oct 29 23:29:46.210588 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.209920 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6" Oct 29 23:29:46.211197 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.211151 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" containerID="cri-o://5caba384c63c1977c4f8c8158c9c364d4df55f4de2634df1cf5671fb7813cf9d" gracePeriod=45 Oct 29 23:29:46.227989 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.227947 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" containerID="cri-o://7506fa9074430cad8569770fb2b15c1500e5f41e87d1f9cf592f52bd91566bc6" gracePeriod=45 Oct 29 23:29:46.521628 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.521589 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:29:46.567262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.567228 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:29:46.652411 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.652378 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-model-cache\") pod \"8372b3e4-533f-4edc-a90c-91b608494d18\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " Oct 29 23:29:46.652658 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.652441 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-dshm\") pod \"8372b3e4-533f-4edc-a90c-91b608494d18\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " Oct 29 23:29:46.652658 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.652505 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-home\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-home\") pod \"8372b3e4-533f-4edc-a90c-91b608494d18\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " Oct 29 23:29:46.652658 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.652583 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8372b3e4-533f-4edc-a90c-91b608494d18-kube-api-access-tfq5p\" (UniqueName: \"kubernetes.io/projected/8372b3e4-533f-4edc-a90c-91b608494d18-kube-api-access-tfq5p\") pod \"8372b3e4-533f-4edc-a90c-91b608494d18\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " Oct 29 23:29:46.652832 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.652678 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-tmp-dir\") pod \"8372b3e4-533f-4edc-a90c-91b608494d18\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " Oct 29 23:29:46.652832 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.652721 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8372b3e4-533f-4edc-a90c-91b608494d18-tls-certs\" (UniqueName: \"kubernetes.io/secret/8372b3e4-533f-4edc-a90c-91b608494d18-tls-certs\") pod \"8372b3e4-533f-4edc-a90c-91b608494d18\" (UID: \"8372b3e4-533f-4edc-a90c-91b608494d18\") " Oct 29 23:29:46.652832 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.652775 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-model-cache" pod "8372b3e4-533f-4edc-a90c-91b608494d18" (UID: "8372b3e4-533f-4edc-a90c-91b608494d18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:46.653502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.653123 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-home" pod "8372b3e4-533f-4edc-a90c-91b608494d18" (UID: "8372b3e4-533f-4edc-a90c-91b608494d18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:46.653502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.653201 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-tmp-dir" pod "8372b3e4-533f-4edc-a90c-91b608494d18" (UID: "8372b3e4-533f-4edc-a90c-91b608494d18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:46.653738 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.653672 6507 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-model-cache\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:46.653738 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.653700 6507 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-home\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:46.653738 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.653714 6507 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-tmp-dir\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:46.656505 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.656472 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8372b3e4-533f-4edc-a90c-91b608494d18-tls-certs" pod "8372b3e4-533f-4edc-a90c-91b608494d18" (UID: "8372b3e4-533f-4edc-a90c-91b608494d18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:46.656612 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.656523 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8372b3e4-533f-4edc-a90c-91b608494d18-kube-api-access-tfq5p" pod "8372b3e4-533f-4edc-a90c-91b608494d18" (UID: "8372b3e4-533f-4edc-a90c-91b608494d18"). InnerVolumeSpecName "kube-api-access-tfq5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:46.657586 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.657560 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-dshm" pod "8372b3e4-533f-4edc-a90c-91b608494d18" (UID: "8372b3e4-533f-4edc-a90c-91b608494d18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:46.755079 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.754994 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-home\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-home\") pod \"1f27f775-181e-4522-b34e-10374fc64c45\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " Oct 29 23:29:46.755079 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.755034 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f27f775-181e-4522-b34e-10374fc64c45-kube-api-access-76xcx\" (UniqueName: \"kubernetes.io/projected/1f27f775-181e-4522-b34e-10374fc64c45-kube-api-access-76xcx\") pod \"1f27f775-181e-4522-b34e-10374fc64c45\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " Oct 29 23:29:46.755305 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.755082 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-tmp-dir\") pod \"1f27f775-181e-4522-b34e-10374fc64c45\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " Oct 29 23:29:46.755305 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.755102 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-dshm\") pod \"1f27f775-181e-4522-b34e-10374fc64c45\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " Oct 29 23:29:46.755305 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.755121 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f27f775-181e-4522-b34e-10374fc64c45-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f27f775-181e-4522-b34e-10374fc64c45-tls-certs\") pod \"1f27f775-181e-4522-b34e-10374fc64c45\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " Oct 29 23:29:46.755305 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.755218 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-model-cache\") pod \"1f27f775-181e-4522-b34e-10374fc64c45\" (UID: \"1f27f775-181e-4522-b34e-10374fc64c45\") " Oct 29 23:29:46.755529 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.755284 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-home" pod "1f27f775-181e-4522-b34e-10374fc64c45" (UID: "1f27f775-181e-4522-b34e-10374fc64c45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:46.755529 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.755377 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-tmp-dir" pod "1f27f775-181e-4522-b34e-10374fc64c45" (UID: "1f27f775-181e-4522-b34e-10374fc64c45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:46.755628 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.755581 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-model-cache" pod "1f27f775-181e-4522-b34e-10374fc64c45" (UID: "1f27f775-181e-4522-b34e-10374fc64c45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:46.755894 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.755867 6507 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-tmp-dir\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:46.755894 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.755895 6507 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-model-cache\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:46.756089 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.755908 6507 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8372b3e4-533f-4edc-a90c-91b608494d18-dshm\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:46.756089 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.755923 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfq5p\" (UniqueName: \"kubernetes.io/projected/8372b3e4-533f-4edc-a90c-91b608494d18-kube-api-access-tfq5p\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:46.756089 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.755937 6507 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8372b3e4-533f-4edc-a90c-91b608494d18-tls-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:46.756089 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.755950 6507 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-home\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:46.757305 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.757278 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-dshm" pod "1f27f775-181e-4522-b34e-10374fc64c45" (UID: "1f27f775-181e-4522-b34e-10374fc64c45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:46.757424 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.757400 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f27f775-181e-4522-b34e-10374fc64c45-kube-api-access-76xcx" pod "1f27f775-181e-4522-b34e-10374fc64c45" (UID: "1f27f775-181e-4522-b34e-10374fc64c45"). InnerVolumeSpecName "kube-api-access-76xcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:46.757500 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.757468 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f27f775-181e-4522-b34e-10374fc64c45-tls-certs" pod "1f27f775-181e-4522-b34e-10374fc64c45" (UID: "1f27f775-181e-4522-b34e-10374fc64c45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:46.857964 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.857929 6507 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f27f775-181e-4522-b34e-10374fc64c45-dshm\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:46.857964 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.857961 6507 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f27f775-181e-4522-b34e-10374fc64c45-tls-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:46.857964 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:46.857976 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76xcx\" (UniqueName: \"kubernetes.io/projected/1f27f775-181e-4522-b34e-10374fc64c45-kube-api-access-76xcx\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:47.194970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.194934 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq"] Oct 29 23:29:47.195627 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.195611 6507 cpu_manager.go:414] "containerMap: removing container" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" Oct 29 23:29:47.195691 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.195630 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" Oct 29 23:29:47.195691 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.195649 6507 cpu_manager.go:414] "containerMap: removing container" podUID="e69bec15-7520-4331-a8af-bf1cefd3fd1e" containerName="revoke-keys" Oct 29 23:29:47.195691 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.195654 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e69bec15-7520-4331-a8af-bf1cefd3fd1e" containerName="revoke-keys" Oct 29 23:29:47.195691 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.195681 6507 cpu_manager.go:414] "containerMap: removing container" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" Oct 29 23:29:47.195691 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.195689 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" Oct 29 23:29:47.195841 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.195700 6507 cpu_manager.go:414] "containerMap: removing container" podUID="60bf6b38-c0c8-4e8b-bd72-6876edf4dae3" containerName="revoke-keys" Oct 29 23:29:47.195841 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.195706 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60bf6b38-c0c8-4e8b-bd72-6876edf4dae3" containerName="revoke-keys" Oct 29 23:29:47.195841 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.195819 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" containerName="main" Oct 29 23:29:47.195841 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.195830 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="60bf6b38-c0c8-4e8b-bd72-6876edf4dae3" containerName="revoke-keys" Oct 29 23:29:47.195841 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.195840 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f27f775-181e-4522-b34e-10374fc64c45" containerName="main" Oct 29 23:29:47.195996 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.195848 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="e69bec15-7520-4331-a8af-bf1cefd3fd1e" containerName="revoke-keys" Oct 29 23:29:47.201241 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.201224 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq" Oct 29 23:29:47.220574 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.220531 6507 generic.go:356] "Generic (PLEG): container finished" podID="8372b3e4-533f-4edc-a90c-91b608494d18" containerID="5caba384c63c1977c4f8c8158c9c364d4df55f4de2634df1cf5671fb7813cf9d" exitCode=0 Oct 29 23:29:47.220959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.220640 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" event={"ID":"8372b3e4-533f-4edc-a90c-91b608494d18","Type":"ContainerDied","Data":"5caba384c63c1977c4f8c8158c9c364d4df55f4de2634df1cf5671fb7813cf9d"} Oct 29 23:29:47.220959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.220672 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" event={"ID":"8372b3e4-533f-4edc-a90c-91b608494d18","Type":"ContainerDied","Data":"bd6a93e572aed66cd2c9ad81b4d2fc0266c4900dbbd05578ab89a9f03a214743"} Oct 29 23:29:47.220959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.220692 6507 scope.go:122] "RemoveContainer" containerID="5caba384c63c1977c4f8c8158c9c364d4df55f4de2634df1cf5671fb7813cf9d" Oct 29 23:29:47.220959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.220831 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx" Oct 29 23:29:47.224511 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.224485 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:29:47.224649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.224633 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:29:47.224733 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.224719 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:29:47.224795 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.224780 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:29:47.224841 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.224791 6507 generic.go:356] "Generic (PLEG): container finished" podID="1f27f775-181e-4522-b34e-10374fc64c45" containerID="7506fa9074430cad8569770fb2b15c1500e5f41e87d1f9cf592f52bd91566bc6" exitCode=0 Oct 29 23:29:47.224841 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.224822 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" event={"ID":"1f27f775-181e-4522-b34e-10374fc64c45","Type":"ContainerDied","Data":"7506fa9074430cad8569770fb2b15c1500e5f41e87d1f9cf592f52bd91566bc6"} Oct 29 23:29:47.224937 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.224852 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" event={"ID":"1f27f775-181e-4522-b34e-10374fc64c45","Type":"ContainerDied","Data":"b79160fadbc65ba3c4f403c62f37338785a779cc0c02635a32be6114b5f022b0"} Oct 29 23:29:47.224937 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.224869 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc" Oct 29 23:29:47.231981 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.231946 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq"] Oct 29 23:29:47.239031 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.238912 6507 scope.go:122] "RemoveContainer" containerID="5caba384c63c1977c4f8c8158c9c364d4df55f4de2634df1cf5671fb7813cf9d" Oct 29 23:29:47.239586 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:29:47.239419 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5caba384c63c1977c4f8c8158c9c364d4df55f4de2634df1cf5671fb7813cf9d\": container with ID starting with 5caba384c63c1977c4f8c8158c9c364d4df55f4de2634df1cf5671fb7813cf9d not found: ID does not exist" containerID="5caba384c63c1977c4f8c8158c9c364d4df55f4de2634df1cf5671fb7813cf9d" Oct 29 23:29:47.239586 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.239497 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5caba384c63c1977c4f8c8158c9c364d4df55f4de2634df1cf5671fb7813cf9d"} err="failed to get container status \"5caba384c63c1977c4f8c8158c9c364d4df55f4de2634df1cf5671fb7813cf9d\": rpc error: code = NotFound desc = could not find container \"5caba384c63c1977c4f8c8158c9c364d4df55f4de2634df1cf5671fb7813cf9d\": container with ID starting with 5caba384c63c1977c4f8c8158c9c364d4df55f4de2634df1cf5671fb7813cf9d not found: ID does not exist" Oct 29 23:29:47.239586 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.239520 6507 scope.go:122] "RemoveContainer" containerID="7506fa9074430cad8569770fb2b15c1500e5f41e87d1f9cf592f52bd91566bc6" Oct 29 23:29:47.255568 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.255540 6507 scope.go:122] "RemoveContainer" containerID="7506fa9074430cad8569770fb2b15c1500e5f41e87d1f9cf592f52bd91566bc6" Oct 29 23:29:47.256341 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:29:47.256319 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7506fa9074430cad8569770fb2b15c1500e5f41e87d1f9cf592f52bd91566bc6\": container with ID starting with 7506fa9074430cad8569770fb2b15c1500e5f41e87d1f9cf592f52bd91566bc6 not found: ID does not exist" containerID="7506fa9074430cad8569770fb2b15c1500e5f41e87d1f9cf592f52bd91566bc6" Oct 29 23:29:47.256425 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.256354 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7506fa9074430cad8569770fb2b15c1500e5f41e87d1f9cf592f52bd91566bc6"} err="failed to get container status \"7506fa9074430cad8569770fb2b15c1500e5f41e87d1f9cf592f52bd91566bc6\": rpc error: code = NotFound desc = could not find container \"7506fa9074430cad8569770fb2b15c1500e5f41e87d1f9cf592f52bd91566bc6\": container with ID starting with 7506fa9074430cad8569770fb2b15c1500e5f41e87d1f9cf592f52bd91566bc6 not found: ID does not exist" Oct 29 23:29:47.288758 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.288683 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx"] Oct 29 23:29:47.294596 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.294563 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-313d9c/distinct-2-w1-313d9c-kserve-658dc99b96-m2jgx"] Oct 29 23:29:47.325586 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.325553 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc"] Oct 29 23:29:47.329684 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.329660 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-313d9c/simulator-w1-313d9c-kserve-6bdbdf68df-lq6lc"] Oct 29 23:29:47.365528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.365494 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21406741-12e6-4996-95b1-23be60d44ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50d79429-vfqwq\" (UID: \"21406741-12e6-4996-95b1-23be60d44ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq" Oct 29 23:29:47.365855 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.365830 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfjc5\" (UniqueName: \"kubernetes.io/projected/21406741-12e6-4996-95b1-23be60d44ee1-kube-api-access-gfjc5\") pod \"maas-api-revoke-sub-50d79429-vfqwq\" (UID: \"21406741-12e6-4996-95b1-23be60d44ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq" Oct 29 23:29:47.365959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.365940 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21406741-12e6-4996-95b1-23be60d44ee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50d79429-vfqwq\" (UID: \"21406741-12e6-4996-95b1-23be60d44ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq" Oct 29 23:29:47.477592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.472307 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfjc5\" (UniqueName: \"kubernetes.io/projected/21406741-12e6-4996-95b1-23be60d44ee1-kube-api-access-gfjc5\") pod \"maas-api-revoke-sub-50d79429-vfqwq\" (UID: \"21406741-12e6-4996-95b1-23be60d44ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq" Oct 29 23:29:47.477592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.472436 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21406741-12e6-4996-95b1-23be60d44ee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50d79429-vfqwq\" (UID: \"21406741-12e6-4996-95b1-23be60d44ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq" Oct 29 23:29:47.477592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.472518 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21406741-12e6-4996-95b1-23be60d44ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50d79429-vfqwq\" (UID: \"21406741-12e6-4996-95b1-23be60d44ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq" Oct 29 23:29:47.481487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.480149 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21406741-12e6-4996-95b1-23be60d44ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50d79429-vfqwq\" (UID: \"21406741-12e6-4996-95b1-23be60d44ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq" Oct 29 23:29:47.511602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.507650 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfjc5\" (UniqueName: \"kubernetes.io/projected/21406741-12e6-4996-95b1-23be60d44ee1-kube-api-access-gfjc5\") pod \"maas-api-revoke-sub-50d79429-vfqwq\" (UID: \"21406741-12e6-4996-95b1-23be60d44ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq" Oct 29 23:29:47.533450 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.522938 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21406741-12e6-4996-95b1-23be60d44ee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50d79429-vfqwq\" (UID: \"21406741-12e6-4996-95b1-23be60d44ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq" Oct 29 23:29:47.816559 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:47.816451 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq" Oct 29 23:29:48.126870 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:48.126835 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq"] Oct 29 23:29:48.128070 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:29:48.128044 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21406741_12e6_4996_95b1_23be60d44ee1.slice/crio-29360cde9cb843e730cfc89c096fe29b8037d2fb0dbb2169f42c8d23de7b7113 WatchSource:0}: Error finding container 29360cde9cb843e730cfc89c096fe29b8037d2fb0dbb2169f42c8d23de7b7113: Status 404 returned error can't find the container with id 29360cde9cb843e730cfc89c096fe29b8037d2fb0dbb2169f42c8d23de7b7113 Oct 29 23:29:48.237133 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:48.237088 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq" event={"ID":"21406741-12e6-4996-95b1-23be60d44ee1","Type":"ContainerStarted","Data":"29360cde9cb843e730cfc89c096fe29b8037d2fb0dbb2169f42c8d23de7b7113"} Oct 29 23:29:48.754956 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:48.754918 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f27f775-181e-4522-b34e-10374fc64c45" path="/var/lib/kubelet/pods/1f27f775-181e-4522-b34e-10374fc64c45/volumes" Oct 29 23:29:48.755431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:48.755417 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8372b3e4-533f-4edc-a90c-91b608494d18" path="/var/lib/kubelet/pods/8372b3e4-533f-4edc-a90c-91b608494d18/volumes" Oct 29 23:29:49.247383 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:49.247349 6507 generic.go:356] "Generic (PLEG): container finished" podID="21406741-12e6-4996-95b1-23be60d44ee1" containerID="5295b915ce26580b1e008566ede142f9dd99d763edc096f2c3062c3c4137d298" exitCode=0 Oct 29 23:29:49.247794 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:49.247436 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq" event={"ID":"21406741-12e6-4996-95b1-23be60d44ee1","Type":"ContainerDied","Data":"5295b915ce26580b1e008566ede142f9dd99d763edc096f2c3062c3c4137d298"} Oct 29 23:29:49.489447 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:49.489412 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" podUID="6173dedb-87b4-4028-9506-34ffadb93263" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:29:50.520269 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:50.519958 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" podUID="9c730892-1170-44b8-9fcf-e74a43035722" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:29:51.747678 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:51.747555 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 29 23:29:51.846400 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:29:51.820669 6507 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.075s" Oct 29 23:29:51.873637 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:51.873590 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:51.890487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:51.881901 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded" Oct 29 23:29:52.242484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.238754 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq" Oct 29 23:29:52.276562 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.276325 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21406741-12e6-4996-95b1-23be60d44ee1-kube-api-access-gfjc5\" (UniqueName: \"kubernetes.io/projected/21406741-12e6-4996-95b1-23be60d44ee1-kube-api-access-gfjc5\") pod \"21406741-12e6-4996-95b1-23be60d44ee1\" (UID: \"21406741-12e6-4996-95b1-23be60d44ee1\") " Oct 29 23:29:52.277357 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.276876 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21406741-12e6-4996-95b1-23be60d44ee1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21406741-12e6-4996-95b1-23be60d44ee1-maas-api-xks-ca\") pod \"21406741-12e6-4996-95b1-23be60d44ee1\" (UID: \"21406741-12e6-4996-95b1-23be60d44ee1\") " Oct 29 23:29:52.277357 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.277020 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21406741-12e6-4996-95b1-23be60d44ee1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21406741-12e6-4996-95b1-23be60d44ee1-maas-api-openshift-ca\") pod \"21406741-12e6-4996-95b1-23be60d44ee1\" (UID: \"21406741-12e6-4996-95b1-23be60d44ee1\") " Oct 29 23:29:52.289526 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.289484 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21406741-12e6-4996-95b1-23be60d44ee1-maas-api-openshift-ca" pod "21406741-12e6-4996-95b1-23be60d44ee1" (UID: "21406741-12e6-4996-95b1-23be60d44ee1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:52.311487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.300643 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21406741-12e6-4996-95b1-23be60d44ee1-kube-api-access-gfjc5" pod "21406741-12e6-4996-95b1-23be60d44ee1" (UID: "21406741-12e6-4996-95b1-23be60d44ee1"). InnerVolumeSpecName "kube-api-access-gfjc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:52.311487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.302089 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21406741-12e6-4996-95b1-23be60d44ee1-maas-api-xks-ca" pod "21406741-12e6-4996-95b1-23be60d44ee1" (UID: "21406741-12e6-4996-95b1-23be60d44ee1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:52.311487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.310139 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21406741-12e6-4996-95b1-23be60d44ee1-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:52.311487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.310167 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21406741-12e6-4996-95b1-23be60d44ee1-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:52.311487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.310183 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfjc5\" (UniqueName: \"kubernetes.io/projected/21406741-12e6-4996-95b1-23be60d44ee1-kube-api-access-gfjc5\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:52.360240 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.360191 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:52.478483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.472559 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:52.524502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.524430 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:52.788483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.787770 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:52.788483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.788449 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:29:52.853436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.853402 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:29:52.874737 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.874697 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 23:29:52.890195 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:52.890159 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:29:53.178543 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.176902 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 29 23:29:53.252478 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.224483 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6"] Oct 29 23:29:53.252478 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.225538 6507 cpu_manager.go:414] "containerMap: removing container" podUID="21406741-12e6-4996-95b1-23be60d44ee1" containerName="revoke-keys" Oct 29 23:29:53.252478 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.225561 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21406741-12e6-4996-95b1-23be60d44ee1" containerName="revoke-keys" Oct 29 23:29:53.252478 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.225750 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="21406741-12e6-4996-95b1-23be60d44ee1" containerName="revoke-keys" Oct 29 23:29:53.357498 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.353961 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6"] Oct 29 23:29:53.357498 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.354104 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" Oct 29 23:29:53.380948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.379857 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:53.380948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.380234 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 23:29:53.380948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.380948 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:29:53.440547 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.437615 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:53.502690 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.469777 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxn42\" (UniqueName: \"kubernetes.io/projected/d767e464-9ffc-412c-a15f-f815f7a4caa0-kube-api-access-hxn42\") pod \"maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6\" (UID: \"d767e464-9ffc-412c-a15f-f815f7a4caa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" Oct 29 23:29:53.503032 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.502998 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d767e464-9ffc-412c-a15f-f815f7a4caa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6\" (UID: \"d767e464-9ffc-412c-a15f-f815f7a4caa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" Oct 29 23:29:53.503217 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.503192 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d767e464-9ffc-412c-a15f-f815f7a4caa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6\" (UID: \"d767e464-9ffc-412c-a15f-f815f7a4caa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" Oct 29 23:29:53.503850 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.503817 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:53.504734 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.504714 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:29:53.542707 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.542656 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq" event={"ID":"21406741-12e6-4996-95b1-23be60d44ee1","Type":"ContainerDied","Data":"29360cde9cb843e730cfc89c096fe29b8037d2fb0dbb2169f42c8d23de7b7113"} Oct 29 23:29:53.571823 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.571785 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29360cde9cb843e730cfc89c096fe29b8037d2fb0dbb2169f42c8d23de7b7113" Oct 29 23:29:53.572985 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.543545 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:53.591652 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.591617 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:29:53.596594 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.596550 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq" Oct 29 23:29:53.655593 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.655552 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxn42\" (UniqueName: \"kubernetes.io/projected/d767e464-9ffc-412c-a15f-f815f7a4caa0-kube-api-access-hxn42\") pod \"maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6\" (UID: \"d767e464-9ffc-412c-a15f-f815f7a4caa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" Oct 29 23:29:53.711980 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.711882 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d767e464-9ffc-412c-a15f-f815f7a4caa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6\" (UID: \"d767e464-9ffc-412c-a15f-f815f7a4caa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" Oct 29 23:29:53.712230 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.712209 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d767e464-9ffc-412c-a15f-f815f7a4caa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6\" (UID: \"d767e464-9ffc-412c-a15f-f815f7a4caa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" Oct 29 23:29:53.734503 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.713985 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d767e464-9ffc-412c-a15f-f815f7a4caa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6\" (UID: \"d767e464-9ffc-412c-a15f-f815f7a4caa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" Oct 29 23:29:53.809705 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.809639 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:29:53.810790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.810760 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:29:53.827489 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.812212 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:53.920482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.915088 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:53.920482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:53.918031 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d767e464-9ffc-412c-a15f-f815f7a4caa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6\" (UID: \"d767e464-9ffc-412c-a15f-f815f7a4caa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" Oct 29 23:29:54.142551 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.123647 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:54.184546 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.184490 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:54.185760 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.185735 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:29:54.286481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.282591 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:54.442551 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.419314 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxn42\" (UniqueName: \"kubernetes.io/projected/d767e464-9ffc-412c-a15f-f815f7a4caa0-kube-api-access-hxn42\") pod \"maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6\" (UID: \"d767e464-9ffc-412c-a15f-f815f7a4caa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" Oct 29 23:29:54.442862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.428957 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 23:29:54.464430 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.429001 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 23:29:54.535525 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.465910 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:29:54.535525 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.476919 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:54.535525 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.532857 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:54.536625 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.536591 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:29:54.549686 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.549636 6507 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 29 23:29:54.583089 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.579031 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:29:54.583089 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.580987 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:54.583089 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.580266 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:54.595637 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.595594 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:29:54.595966 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.595924 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" podUID="039b4caf-d11b-4933-af15-2efb408c36e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:54.797548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.776378 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" Oct 29 23:29:54.846046 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.846002 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:54.991531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.976676 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 29 23:29:54.991531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.978209 6507 patch_prober.go:29] interesting pod/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:29:54.991531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:54.978266 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" podUID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:55.016697 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:55.016642 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:55.211483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:55.202560 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:55.298056 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:55.298017 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:29:55.497535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:55.497492 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 23:29:55.612483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:55.541771 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 23:29:55.614568 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:55.614528 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 29 23:29:55.729484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:55.685267 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:55.805282 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:55.805168 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:55.851489 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:55.848822 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:56.011208 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.011162 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:56.196480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.193289 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" podUID="1020c327-0d07-4228-8124-ce2b9d0413ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded" Oct 29 23:29:56.251484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.203742 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:29:56.356487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.348102 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv" Oct 29 23:29:56.408922 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.408889 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:29:56.441653 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.441626 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:29:56.531244 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.531218 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:29:56.535503 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.533215 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:29:56.535503 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.533902 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:29:56.541480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.539626 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk"] Oct 29 23:29:56.543191 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.543170 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:29:56.543364 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.543350 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf"] Oct 29 23:29:56.544916 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.544901 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:29:56.551720 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.549130 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h"] Oct 29 23:29:56.557579 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.556049 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h" Oct 29 23:29:56.561417 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.561373 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h"] Oct 29 23:29:56.680032 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.679983 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ff301fd-wrx5h\" (UID: \"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h" Oct 29 23:29:56.680254 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.680230 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ff301fd-wrx5h\" (UID: \"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h" Oct 29 23:29:56.680338 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.680321 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mmgj\" (UniqueName: \"kubernetes.io/projected/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-kube-api-access-7mmgj\") pod \"maas-api-revoke-sub-7ff301fd-wrx5h\" (UID: \"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h" Oct 29 23:29:56.720894 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.720861 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6"] Oct 29 23:29:56.722544 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:29:56.722516 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd767e464_9ffc_412c_a15f_f815f7a4caa0.slice/crio-f877e1254427dddce9b634ecb54e6a9a2a02e88dc4599f2936c2f2afe4d2221c WatchSource:0}: Error finding container f877e1254427dddce9b634ecb54e6a9a2a02e88dc4599f2936c2f2afe4d2221c: Status 404 returned error can't find the container with id f877e1254427dddce9b634ecb54e6a9a2a02e88dc4599f2936c2f2afe4d2221c Oct 29 23:29:56.781370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.781332 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mmgj\" (UniqueName: \"kubernetes.io/projected/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-kube-api-access-7mmgj\") pod \"maas-api-revoke-sub-7ff301fd-wrx5h\" (UID: \"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h" Oct 29 23:29:56.781713 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.781562 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ff301fd-wrx5h\" (UID: \"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h" Oct 29 23:29:56.781822 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.781773 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ff301fd-wrx5h\" (UID: \"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h" Oct 29 23:29:56.782392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.782369 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ff301fd-wrx5h\" (UID: \"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h" Oct 29 23:29:56.784277 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.784252 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ff301fd-wrx5h\" (UID: \"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h" Oct 29 23:29:56.798552 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.798522 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mmgj\" (UniqueName: \"kubernetes.io/projected/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-kube-api-access-7mmgj\") pod \"maas-api-revoke-sub-7ff301fd-wrx5h\" (UID: \"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h" Oct 29 23:29:56.877560 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:56.877527 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h" Oct 29 23:29:57.191352 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:57.191327 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h"] Oct 29 23:29:57.192667 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:29:57.192642 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87da7dbd_9b2c_4cc4_98c9_1a302d52ce68.slice/crio-613a9e38ce01b99939644cbe25aff54695e34f5518e6b23abc5ab3913b08298a WatchSource:0}: Error finding container 613a9e38ce01b99939644cbe25aff54695e34f5518e6b23abc5ab3913b08298a: Status 404 returned error can't find the container with id 613a9e38ce01b99939644cbe25aff54695e34f5518e6b23abc5ab3913b08298a Oct 29 23:29:57.393007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:57.392922 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" event={"ID":"d767e464-9ffc-412c-a15f-f815f7a4caa0","Type":"ContainerStarted","Data":"367c8108bda185276353053b9b26e69d7aa717f04968b82982b4c4ac99c6d149"} Oct 29 23:29:57.393007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:57.392979 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" event={"ID":"d767e464-9ffc-412c-a15f-f815f7a4caa0","Type":"ContainerStarted","Data":"f877e1254427dddce9b634ecb54e6a9a2a02e88dc4599f2936c2f2afe4d2221c"} Oct 29 23:29:57.394334 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:57.394310 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h" event={"ID":"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68","Type":"ContainerStarted","Data":"613a9e38ce01b99939644cbe25aff54695e34f5518e6b23abc5ab3913b08298a"} Oct 29 23:29:57.427704 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:57.427649 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" podStartSLOduration=5.427632919 podStartE2EDuration="5.427632919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:57.424682975 +0000 UTC m=+966.787687740" watchObservedRunningTime="2026-10-29 23:29:57.427632919 +0000 UTC m=+966.790637644" Oct 29 23:29:58.403919 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:58.403837 6507 generic.go:356] "Generic (PLEG): container finished" podID="d767e464-9ffc-412c-a15f-f815f7a4caa0" containerID="367c8108bda185276353053b9b26e69d7aa717f04968b82982b4c4ac99c6d149" exitCode=0 Oct 29 23:29:58.404365 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:58.403916 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" event={"ID":"d767e464-9ffc-412c-a15f-f815f7a4caa0","Type":"ContainerDied","Data":"367c8108bda185276353053b9b26e69d7aa717f04968b82982b4c4ac99c6d149"} Oct 29 23:29:58.405953 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:58.405931 6507 generic.go:356] "Generic (PLEG): container finished" podID="87da7dbd-9b2c-4cc4-98c9-1a302d52ce68" containerID="14405dd609be90b271fed73afb5b079e590a5ea82518ed060c3ca30390e01548" exitCode=0 Oct 29 23:29:58.406066 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:58.406015 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h" event={"ID":"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68","Type":"ContainerDied","Data":"14405dd609be90b271fed73afb5b079e590a5ea82518ed060c3ca30390e01548"} Oct 29 23:29:59.495329 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.495293 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" podUID="6173dedb-87b4-4028-9506-34ffadb93263" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:29:59.669723 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.669161 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h" Oct 29 23:29:59.751808 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.751777 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" Oct 29 23:29:59.769501 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.769421 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-maas-api-xks-ca\") pod \"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68\" (UID: \"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68\") " Oct 29 23:29:59.769701 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.769508 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-maas-api-openshift-ca\") pod \"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68\" (UID: \"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68\") " Oct 29 23:29:59.769701 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.769628 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-kube-api-access-7mmgj\" (UniqueName: \"kubernetes.io/projected/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-kube-api-access-7mmgj\") pod \"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68\" (UID: \"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68\") " Oct 29 23:29:59.769930 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.769899 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-maas-api-openshift-ca" pod "87da7dbd-9b2c-4cc4-98c9-1a302d52ce68" (UID: "87da7dbd-9b2c-4cc4-98c9-1a302d52ce68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:59.770395 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.770371 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:59.771810 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.771780 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-kube-api-access-7mmgj" pod "87da7dbd-9b2c-4cc4-98c9-1a302d52ce68" (UID: "87da7dbd-9b2c-4cc4-98c9-1a302d52ce68"). InnerVolumeSpecName "kube-api-access-7mmgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:59.772833 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.772806 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-maas-api-xks-ca" pod "87da7dbd-9b2c-4cc4-98c9-1a302d52ce68" (UID: "87da7dbd-9b2c-4cc4-98c9-1a302d52ce68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:59.871508 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.871443 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d767e464-9ffc-412c-a15f-f815f7a4caa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d767e464-9ffc-412c-a15f-f815f7a4caa0-maas-api-openshift-ca\") pod \"d767e464-9ffc-412c-a15f-f815f7a4caa0\" (UID: \"d767e464-9ffc-412c-a15f-f815f7a4caa0\") " Oct 29 23:29:59.871677 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.871522 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d767e464-9ffc-412c-a15f-f815f7a4caa0-kube-api-access-hxn42\" (UniqueName: \"kubernetes.io/projected/d767e464-9ffc-412c-a15f-f815f7a4caa0-kube-api-access-hxn42\") pod \"d767e464-9ffc-412c-a15f-f815f7a4caa0\" (UID: \"d767e464-9ffc-412c-a15f-f815f7a4caa0\") " Oct 29 23:29:59.871677 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.871588 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d767e464-9ffc-412c-a15f-f815f7a4caa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d767e464-9ffc-412c-a15f-f815f7a4caa0-maas-api-xks-ca\") pod \"d767e464-9ffc-412c-a15f-f815f7a4caa0\" (UID: \"d767e464-9ffc-412c-a15f-f815f7a4caa0\") " Oct 29 23:29:59.871856 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.871832 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d767e464-9ffc-412c-a15f-f815f7a4caa0-maas-api-openshift-ca" pod "d767e464-9ffc-412c-a15f-f815f7a4caa0" (UID: "d767e464-9ffc-412c-a15f-f815f7a4caa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:59.872286 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.872267 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:59.872369 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.872288 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d767e464-9ffc-412c-a15f-f815f7a4caa0-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:59.872369 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.872309 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mmgj\" (UniqueName: \"kubernetes.io/projected/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68-kube-api-access-7mmgj\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:59.873696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.873672 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d767e464-9ffc-412c-a15f-f815f7a4caa0-kube-api-access-hxn42" pod "d767e464-9ffc-412c-a15f-f815f7a4caa0" (UID: "d767e464-9ffc-412c-a15f-f815f7a4caa0"). InnerVolumeSpecName "kube-api-access-hxn42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:59.873764 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.873689 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d767e464-9ffc-412c-a15f-f815f7a4caa0-maas-api-xks-ca" pod "d767e464-9ffc-412c-a15f-f815f7a4caa0" (UID: "d767e464-9ffc-412c-a15f-f815f7a4caa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:59.974103 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.974068 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxn42\" (UniqueName: \"kubernetes.io/projected/d767e464-9ffc-412c-a15f-f815f7a4caa0-kube-api-access-hxn42\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:29:59.974103 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:29:59.974100 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d767e464-9ffc-412c-a15f-f815f7a4caa0-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:00.162399 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.162359 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg"] Oct 29 23:30:00.163982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.163938 6507 cpu_manager.go:414] "containerMap: removing container" podUID="d767e464-9ffc-412c-a15f-f815f7a4caa0" containerName="revoke-keys" Oct 29 23:30:00.163982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.163965 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d767e464-9ffc-412c-a15f-f815f7a4caa0" containerName="revoke-keys" Oct 29 23:30:00.164202 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.164027 6507 cpu_manager.go:414] "containerMap: removing container" podUID="87da7dbd-9b2c-4cc4-98c9-1a302d52ce68" containerName="revoke-keys" Oct 29 23:30:00.164202 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.164035 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87da7dbd-9b2c-4cc4-98c9-1a302d52ce68" containerName="revoke-keys" Oct 29 23:30:00.164202 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.164178 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="87da7dbd-9b2c-4cc4-98c9-1a302d52ce68" containerName="revoke-keys" Oct 29 23:30:00.164202 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.164196 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="d767e464-9ffc-412c-a15f-f815f7a4caa0" containerName="revoke-keys" Oct 29 23:30:00.168340 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.168315 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" Oct 29 23:30:00.168860 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.168837 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j"] Oct 29 23:30:00.173079 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.173062 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" Oct 29 23:30:00.173359 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.173336 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq"] Oct 29 23:30:00.178679 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.178649 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" Oct 29 23:30:00.194571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.194525 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz"] Oct 29 23:30:00.208835 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.208804 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj"] Oct 29 23:30:00.209010 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.208946 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" Oct 29 23:30:00.221857 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.216654 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg"] Oct 29 23:30:00.221857 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.216810 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" Oct 29 23:30:00.221857 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.216979 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq"] Oct 29 23:30:00.224650 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.223900 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j"] Oct 29 23:30:00.239311 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.239240 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz"] Oct 29 23:30:00.245615 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.245585 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj"] Oct 29 23:30:00.278234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.278193 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185d4120-7013-49b4-af2d-0ec8aa4a7f14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j\" (UID: \"185d4120-7013-49b4-af2d-0ec8aa4a7f14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" Oct 29 23:30:00.278430 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.278325 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n28g\" (UniqueName: \"kubernetes.io/projected/185d4120-7013-49b4-af2d-0ec8aa4a7f14-kube-api-access-7n28g\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j\" (UID: \"185d4120-7013-49b4-af2d-0ec8aa4a7f14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" Oct 29 23:30:00.278430 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.278362 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdw7h\" (UniqueName: \"kubernetes.io/projected/fa54faa3-3bb0-477e-9e21-a44b1d5a92b6-kube-api-access-mdw7h\") pod \"maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg\" (UID: \"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" Oct 29 23:30:00.278667 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.278518 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa54faa3-3bb0-477e-9e21-a44b1d5a92b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg\" (UID: \"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" Oct 29 23:30:00.278667 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.278581 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a417439-524a-4ee1-ab88-3f083c09c179-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz\" (UID: \"3a417439-524a-4ee1-ab88-3f083c09c179\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" Oct 29 23:30:00.278786 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.278757 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd068bac-7513-4e03-ab75-5f6777222571-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888610-pk9cq\" (UID: \"dd068bac-7513-4e03-ab75-5f6777222571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" Oct 29 23:30:00.278842 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.278815 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcdbh\" (UniqueName: \"kubernetes.io/projected/dd068bac-7513-4e03-ab75-5f6777222571-kube-api-access-pcdbh\") pod \"maas-api-key-cleanup-29888610-pk9cq\" (UID: \"dd068bac-7513-4e03-ab75-5f6777222571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" Oct 29 23:30:00.278934 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.278906 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl7gc\" (UniqueName: \"kubernetes.io/projected/3a417439-524a-4ee1-ab88-3f083c09c179-kube-api-access-dl7gc\") pod \"maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz\" (UID: \"3a417439-524a-4ee1-ab88-3f083c09c179\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" Oct 29 23:30:00.380845 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.380808 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n28g\" (UniqueName: \"kubernetes.io/projected/185d4120-7013-49b4-af2d-0ec8aa4a7f14-kube-api-access-7n28g\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j\" (UID: \"185d4120-7013-49b4-af2d-0ec8aa4a7f14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" Oct 29 23:30:00.381082 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.380859 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdw7h\" (UniqueName: \"kubernetes.io/projected/fa54faa3-3bb0-477e-9e21-a44b1d5a92b6-kube-api-access-mdw7h\") pod \"maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg\" (UID: \"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" Oct 29 23:30:00.381082 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.380904 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa54faa3-3bb0-477e-9e21-a44b1d5a92b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg\" (UID: \"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" Oct 29 23:30:00.381082 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.380943 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a417439-524a-4ee1-ab88-3f083c09c179-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz\" (UID: \"3a417439-524a-4ee1-ab88-3f083c09c179\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" Oct 29 23:30:00.381082 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.381037 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd068bac-7513-4e03-ab75-5f6777222571-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888610-pk9cq\" (UID: \"dd068bac-7513-4e03-ab75-5f6777222571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" Oct 29 23:30:00.381082 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.381069 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zbzz\" (UniqueName: \"kubernetes.io/projected/5f0f0d26-ae35-48ff-9ee6-d591f159ffcc-kube-api-access-5zbzz\") pod \"maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj\" (UID: \"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" Oct 29 23:30:00.381341 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.381172 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcdbh\" (UniqueName: \"kubernetes.io/projected/dd068bac-7513-4e03-ab75-5f6777222571-kube-api-access-pcdbh\") pod \"maas-api-key-cleanup-29888610-pk9cq\" (UID: \"dd068bac-7513-4e03-ab75-5f6777222571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" Oct 29 23:30:00.381341 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.381292 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f0f0d26-ae35-48ff-9ee6-d591f159ffcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj\" (UID: \"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" Oct 29 23:30:00.381444 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.381402 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl7gc\" (UniqueName: \"kubernetes.io/projected/3a417439-524a-4ee1-ab88-3f083c09c179-kube-api-access-dl7gc\") pod \"maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz\" (UID: \"3a417439-524a-4ee1-ab88-3f083c09c179\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" Oct 29 23:30:00.381586 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.381569 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185d4120-7013-49b4-af2d-0ec8aa4a7f14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j\" (UID: \"185d4120-7013-49b4-af2d-0ec8aa4a7f14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" Oct 29 23:30:00.381783 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.381755 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd068bac-7513-4e03-ab75-5f6777222571-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888610-pk9cq\" (UID: \"dd068bac-7513-4e03-ab75-5f6777222571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" Oct 29 23:30:00.381885 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.381789 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa54faa3-3bb0-477e-9e21-a44b1d5a92b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg\" (UID: \"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" Oct 29 23:30:00.381885 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.381795 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a417439-524a-4ee1-ab88-3f083c09c179-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz\" (UID: \"3a417439-524a-4ee1-ab88-3f083c09c179\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" Oct 29 23:30:00.382292 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.382276 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185d4120-7013-49b4-af2d-0ec8aa4a7f14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j\" (UID: \"185d4120-7013-49b4-af2d-0ec8aa4a7f14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" Oct 29 23:30:00.398212 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.398181 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n28g\" (UniqueName: \"kubernetes.io/projected/185d4120-7013-49b4-af2d-0ec8aa4a7f14-kube-api-access-7n28g\") pod \"maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j\" (UID: \"185d4120-7013-49b4-af2d-0ec8aa4a7f14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" Oct 29 23:30:00.398363 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.398181 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcdbh\" (UniqueName: \"kubernetes.io/projected/dd068bac-7513-4e03-ab75-5f6777222571-kube-api-access-pcdbh\") pod \"maas-api-key-cleanup-29888610-pk9cq\" (UID: \"dd068bac-7513-4e03-ab75-5f6777222571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" Oct 29 23:30:00.398363 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.398343 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl7gc\" (UniqueName: \"kubernetes.io/projected/3a417439-524a-4ee1-ab88-3f083c09c179-kube-api-access-dl7gc\") pod \"maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz\" (UID: \"3a417439-524a-4ee1-ab88-3f083c09c179\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" Oct 29 23:30:00.398363 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.398352 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdw7h\" (UniqueName: \"kubernetes.io/projected/fa54faa3-3bb0-477e-9e21-a44b1d5a92b6-kube-api-access-mdw7h\") pod \"maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg\" (UID: \"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" Oct 29 23:30:00.423074 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.423042 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h" Oct 29 23:30:00.423074 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.423055 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h" event={"ID":"87da7dbd-9b2c-4cc4-98c9-1a302d52ce68","Type":"ContainerDied","Data":"613a9e38ce01b99939644cbe25aff54695e34f5518e6b23abc5ab3913b08298a"} Oct 29 23:30:00.423302 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.423155 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="613a9e38ce01b99939644cbe25aff54695e34f5518e6b23abc5ab3913b08298a" Oct 29 23:30:00.425138 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.425104 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" event={"ID":"d767e464-9ffc-412c-a15f-f815f7a4caa0","Type":"ContainerDied","Data":"f877e1254427dddce9b634ecb54e6a9a2a02e88dc4599f2936c2f2afe4d2221c"} Oct 29 23:30:00.425138 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.425133 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6" Oct 29 23:30:00.425328 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.425157 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f877e1254427dddce9b634ecb54e6a9a2a02e88dc4599f2936c2f2afe4d2221c" Oct 29 23:30:00.483290 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.483193 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zbzz\" (UniqueName: \"kubernetes.io/projected/5f0f0d26-ae35-48ff-9ee6-d591f159ffcc-kube-api-access-5zbzz\") pod \"maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj\" (UID: \"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" Oct 29 23:30:00.483444 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.483289 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f0f0d26-ae35-48ff-9ee6-d591f159ffcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj\" (UID: \"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" Oct 29 23:30:00.484012 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.483990 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f0f0d26-ae35-48ff-9ee6-d591f159ffcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj\" (UID: \"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" Oct 29 23:30:00.484625 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.484605 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" Oct 29 23:30:00.497946 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.497906 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" podUID="9c730892-1170-44b8-9fcf-e74a43035722" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:30:00.499951 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.499904 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" Oct 29 23:30:00.500951 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.500928 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zbzz\" (UniqueName: \"kubernetes.io/projected/5f0f0d26-ae35-48ff-9ee6-d591f159ffcc-kube-api-access-5zbzz\") pod \"maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj\" (UID: \"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" Oct 29 23:30:00.534042 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.534008 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" Oct 29 23:30:00.549158 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.549113 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" Oct 29 23:30:00.558727 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.558688 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" Oct 29 23:30:00.814542 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:30:00.814508 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa54faa3_3bb0_477e_9e21_a44b1d5a92b6.slice/crio-651c81c98d7bf3b244c8fe70f35da83b4a2a3c33cb7d5afe69e10f70e7a93bc4 WatchSource:0}: Error finding container 651c81c98d7bf3b244c8fe70f35da83b4a2a3c33cb7d5afe69e10f70e7a93bc4: Status 404 returned error can't find the container with id 651c81c98d7bf3b244c8fe70f35da83b4a2a3c33cb7d5afe69e10f70e7a93bc4 Oct 29 23:30:00.820541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.820511 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg"] Oct 29 23:30:00.906345 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.906319 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz"] Oct 29 23:30:00.909239 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:30:00.909204 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a417439_524a_4ee1_ab88_3f083c09c179.slice/crio-a568c3cf4bfbe834db9a5a3f61d0705395d891b8b4fb0486c21020551c1f2606 WatchSource:0}: Error finding container a568c3cf4bfbe834db9a5a3f61d0705395d891b8b4fb0486c21020551c1f2606: Status 404 returned error can't find the container with id a568c3cf4bfbe834db9a5a3f61d0705395d891b8b4fb0486c21020551c1f2606 Oct 29 23:30:00.913518 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.913490 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq"] Oct 29 23:30:00.913938 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:30:00.913914 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd068bac_7513_4e03_ab75_5f6777222571.slice/crio-66e11db3ba8db271079445f3e41d37386021ce6c1ef68c655018bbc73b9a79d2 WatchSource:0}: Error finding container 66e11db3ba8db271079445f3e41d37386021ce6c1ef68c655018bbc73b9a79d2: Status 404 returned error can't find the container with id 66e11db3ba8db271079445f3e41d37386021ce6c1ef68c655018bbc73b9a79d2 Oct 29 23:30:00.922840 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:00.922813 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj"] Oct 29 23:30:00.925213 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:30:00.925191 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0f0d26_ae35_48ff_9ee6_d591f159ffcc.slice/crio-acf3076c0e4e1f665e42cd930ee896a1499b643f6496a88195af29df87573c81 WatchSource:0}: Error finding container acf3076c0e4e1f665e42cd930ee896a1499b643f6496a88195af29df87573c81: Status 404 returned error can't find the container with id acf3076c0e4e1f665e42cd930ee896a1499b643f6496a88195af29df87573c81 Oct 29 23:30:01.029496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:01.029467 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j"] Oct 29 23:30:01.434383 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:01.434352 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" event={"ID":"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6","Type":"ContainerStarted","Data":"cc3a2a3c3ea7c9c95f984e1f2ec4c1440c5d7b1bd485918944358e99c6b7fe5d"} Oct 29 23:30:01.434582 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:01.434413 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" event={"ID":"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6","Type":"ContainerStarted","Data":"651c81c98d7bf3b244c8fe70f35da83b4a2a3c33cb7d5afe69e10f70e7a93bc4"} Oct 29 23:30:01.436097 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:01.436070 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" event={"ID":"3a417439-524a-4ee1-ab88-3f083c09c179","Type":"ContainerStarted","Data":"dcd6e20b29f861d8d62455d74012c468db6a1844ed1aee07f54c5c9aee57df58"} Oct 29 23:30:01.436097 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:01.436117 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" event={"ID":"3a417439-524a-4ee1-ab88-3f083c09c179","Type":"ContainerStarted","Data":"a568c3cf4bfbe834db9a5a3f61d0705395d891b8b4fb0486c21020551c1f2606"} Oct 29 23:30:01.437716 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:01.437692 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" event={"ID":"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc","Type":"ContainerStarted","Data":"954bba2b1259225ed8c700219b8702ae00bc26f0a565ad9731f03ec0fd5e9482"} Oct 29 23:30:01.437830 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:01.437732 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" event={"ID":"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc","Type":"ContainerStarted","Data":"acf3076c0e4e1f665e42cd930ee896a1499b643f6496a88195af29df87573c81"} Oct 29 23:30:01.438887 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:01.438866 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" event={"ID":"185d4120-7013-49b4-af2d-0ec8aa4a7f14","Type":"ContainerStarted","Data":"fe9e2fe9f7bc66250981339a56e3624cc6de969891b026bb2a0fa27918348fa0"} Oct 29 23:30:01.440235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:01.440215 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" event={"ID":"dd068bac-7513-4e03-ab75-5f6777222571","Type":"ContainerStarted","Data":"880c9a4d3f57dda3b742096f2813f5b5ba716a7592f47e3c9bb883af193f212d"} Oct 29 23:30:01.440322 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:01.440249 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" event={"ID":"dd068bac-7513-4e03-ab75-5f6777222571","Type":"ContainerStarted","Data":"66e11db3ba8db271079445f3e41d37386021ce6c1ef68c655018bbc73b9a79d2"} Oct 29 23:30:01.470401 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:01.470338 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" podStartSLOduration=1.470321105 podStartE2EDuration="1.470321105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:01.466802268 +0000 UTC m=+970.829806990" watchObservedRunningTime="2026-10-29 23:30:01.470321105 +0000 UTC m=+970.833325827" Oct 29 23:30:01.504692 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:01.504637 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" podStartSLOduration=1.504621779 podStartE2EDuration="1.504621779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:01.499802931 +0000 UTC m=+970.862807654" watchObservedRunningTime="2026-10-29 23:30:01.504621779 +0000 UTC m=+970.867626545" Oct 29 23:30:01.536555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:01.536502 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" podStartSLOduration=1.536488259 podStartE2EDuration="1.536488259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:01.534818784 +0000 UTC m=+970.897823506" watchObservedRunningTime="2026-10-29 23:30:01.536488259 +0000 UTC m=+970.899492989" Oct 29 23:30:01.576374 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:01.576316 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" podStartSLOduration=1.5763022819999999 podStartE2EDuration="1.576302282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:01.570495063 +0000 UTC m=+970.933499783" watchObservedRunningTime="2026-10-29 23:30:01.576302282 +0000 UTC m=+970.939307005" Oct 29 23:30:02.449757 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:02.449718 6507 generic.go:356] "Generic (PLEG): container finished" podID="3a417439-524a-4ee1-ab88-3f083c09c179" containerID="dcd6e20b29f861d8d62455d74012c468db6a1844ed1aee07f54c5c9aee57df58" exitCode=60 Oct 29 23:30:02.449942 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:02.449805 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" event={"ID":"3a417439-524a-4ee1-ab88-3f083c09c179","Type":"ContainerDied","Data":"dcd6e20b29f861d8d62455d74012c468db6a1844ed1aee07f54c5c9aee57df58"} Oct 29 23:30:02.450101 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:02.450086 6507 scope.go:122] "RemoveContainer" containerID="dcd6e20b29f861d8d62455d74012c468db6a1844ed1aee07f54c5c9aee57df58" Oct 29 23:30:02.451528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:02.451506 6507 generic.go:356] "Generic (PLEG): container finished" podID="5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" containerID="954bba2b1259225ed8c700219b8702ae00bc26f0a565ad9731f03ec0fd5e9482" exitCode=60 Oct 29 23:30:02.451658 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:02.451579 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" event={"ID":"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc","Type":"ContainerDied","Data":"954bba2b1259225ed8c700219b8702ae00bc26f0a565ad9731f03ec0fd5e9482"} Oct 29 23:30:02.451821 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:02.451804 6507 scope.go:122] "RemoveContainer" containerID="954bba2b1259225ed8c700219b8702ae00bc26f0a565ad9731f03ec0fd5e9482" Oct 29 23:30:02.453545 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:02.453520 6507 generic.go:356] "Generic (PLEG): container finished" podID="185d4120-7013-49b4-af2d-0ec8aa4a7f14" containerID="3f657a3ace2b9bfdaf770aaab159fe331d503ee4dcd96864bd84178ecb9a996b" exitCode=60 Oct 29 23:30:02.453663 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:02.453601 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" event={"ID":"185d4120-7013-49b4-af2d-0ec8aa4a7f14","Type":"ContainerDied","Data":"3f657a3ace2b9bfdaf770aaab159fe331d503ee4dcd96864bd84178ecb9a996b"} Oct 29 23:30:02.453843 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:02.453813 6507 scope.go:122] "RemoveContainer" containerID="3f657a3ace2b9bfdaf770aaab159fe331d503ee4dcd96864bd84178ecb9a996b" Oct 29 23:30:02.455492 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:02.455472 6507 generic.go:356] "Generic (PLEG): container finished" podID="dd068bac-7513-4e03-ab75-5f6777222571" containerID="880c9a4d3f57dda3b742096f2813f5b5ba716a7592f47e3c9bb883af193f212d" exitCode=60 Oct 29 23:30:02.455548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:02.455503 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" event={"ID":"dd068bac-7513-4e03-ab75-5f6777222571","Type":"ContainerDied","Data":"880c9a4d3f57dda3b742096f2813f5b5ba716a7592f47e3c9bb883af193f212d"} Oct 29 23:30:02.455741 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:02.455727 6507 scope.go:122] "RemoveContainer" containerID="880c9a4d3f57dda3b742096f2813f5b5ba716a7592f47e3c9bb883af193f212d" Oct 29 23:30:02.457064 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:02.457043 6507 generic.go:356] "Generic (PLEG): container finished" podID="fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" containerID="cc3a2a3c3ea7c9c95f984e1f2ec4c1440c5d7b1bd485918944358e99c6b7fe5d" exitCode=60 Oct 29 23:30:02.457134 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:02.457072 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" event={"ID":"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6","Type":"ContainerDied","Data":"cc3a2a3c3ea7c9c95f984e1f2ec4c1440c5d7b1bd485918944358e99c6b7fe5d"} Oct 29 23:30:02.457277 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:02.457264 6507 scope.go:122] "RemoveContainer" containerID="cc3a2a3c3ea7c9c95f984e1f2ec4c1440c5d7b1bd485918944358e99c6b7fe5d" Oct 29 23:30:03.467356 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:03.467312 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" event={"ID":"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc","Type":"ContainerStarted","Data":"a74780b5a9fb47af836a17734362930ab6f62832cf5544f42ac2857396b0f36e"} Oct 29 23:30:03.469549 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:03.469518 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" event={"ID":"185d4120-7013-49b4-af2d-0ec8aa4a7f14","Type":"ContainerStarted","Data":"e6087de0b92f77b2cf63b74441aa9bb5b2d30281ad898381ed536548c65d1cff"} Oct 29 23:30:03.471502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:03.471470 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" event={"ID":"dd068bac-7513-4e03-ab75-5f6777222571","Type":"ContainerStarted","Data":"999d37caf6495b9c284c2422f743c6678d4f6d5380160922ad5e8991995407a6"} Oct 29 23:30:03.473322 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:03.473296 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" event={"ID":"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6","Type":"ContainerStarted","Data":"2b2054521df19ab97b6c737dc38a94dbd8491db886a8179a97b46ce93c7ce62c"} Oct 29 23:30:03.475231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:03.475208 6507 generic.go:356] "Generic (PLEG): container finished" podID="3a417439-524a-4ee1-ab88-3f083c09c179" containerID="170ad77dd5061f7556909d4ac87e47f624541c9e21187b0cc25688e956b4035f" exitCode=60 Oct 29 23:30:03.475342 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:03.475252 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" event={"ID":"3a417439-524a-4ee1-ab88-3f083c09c179","Type":"ContainerDied","Data":"170ad77dd5061f7556909d4ac87e47f624541c9e21187b0cc25688e956b4035f"} Oct 29 23:30:03.475342 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:03.475291 6507 scope.go:122] "RemoveContainer" containerID="dcd6e20b29f861d8d62455d74012c468db6a1844ed1aee07f54c5c9aee57df58" Oct 29 23:30:03.475560 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:03.475541 6507 scope.go:122] "RemoveContainer" containerID="170ad77dd5061f7556909d4ac87e47f624541c9e21187b0cc25688e956b4035f" Oct 29 23:30:03.475886 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:03.475865 6507 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz_odh-ai-gateway-infra(3a417439-524a-4ee1-ab88-3f083c09c179)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" podUID="3a417439-524a-4ee1-ab88-3f083c09c179" Oct 29 23:30:03.542716 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:03.542679 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr"] Oct 29 23:30:03.576968 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:03.576926 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z"] Oct 29 23:30:03.636303 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:03.636244 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" podStartSLOduration=3.636228248 podStartE2EDuration="3.636228248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.632125485 +0000 UTC m=+972.995130208" watchObservedRunningTime="2026-10-29 23:30:03.636228248 +0000 UTC m=+972.999232971" Oct 29 23:30:04.498865 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.498828 6507 generic.go:356] "Generic (PLEG): container finished" podID="5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" containerID="a74780b5a9fb47af836a17734362930ab6f62832cf5544f42ac2857396b0f36e" exitCode=60 Oct 29 23:30:04.499958 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.499937 6507 scope.go:122] "RemoveContainer" containerID="a74780b5a9fb47af836a17734362930ab6f62832cf5544f42ac2857396b0f36e" Oct 29 23:30:04.501235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.500489 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" event={"ID":"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc","Type":"ContainerDied","Data":"a74780b5a9fb47af836a17734362930ab6f62832cf5544f42ac2857396b0f36e"} Oct 29 23:30:04.501235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.501186 6507 scope.go:122] "RemoveContainer" containerID="954bba2b1259225ed8c700219b8702ae00bc26f0a565ad9731f03ec0fd5e9482" Oct 29 23:30:04.504295 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:04.503570 6507 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj_odh-ai-gateway-infra(5f0f0d26-ae35-48ff-9ee6-d591f159ffcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" podUID="5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" Oct 29 23:30:04.505676 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.505650 6507 generic.go:356] "Generic (PLEG): container finished" podID="185d4120-7013-49b4-af2d-0ec8aa4a7f14" containerID="e6087de0b92f77b2cf63b74441aa9bb5b2d30281ad898381ed536548c65d1cff" exitCode=60 Oct 29 23:30:04.505774 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.505732 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" event={"ID":"185d4120-7013-49b4-af2d-0ec8aa4a7f14","Type":"ContainerDied","Data":"e6087de0b92f77b2cf63b74441aa9bb5b2d30281ad898381ed536548c65d1cff"} Oct 29 23:30:04.506254 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.506228 6507 scope.go:122] "RemoveContainer" containerID="e6087de0b92f77b2cf63b74441aa9bb5b2d30281ad898381ed536548c65d1cff" Oct 29 23:30:04.506679 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:04.506656 6507 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j_odh-ai-gateway-infra(185d4120-7013-49b4-af2d-0ec8aa4a7f14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" podUID="185d4120-7013-49b4-af2d-0ec8aa4a7f14" Oct 29 23:30:04.508519 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.508486 6507 generic.go:356] "Generic (PLEG): container finished" podID="dd068bac-7513-4e03-ab75-5f6777222571" containerID="999d37caf6495b9c284c2422f743c6678d4f6d5380160922ad5e8991995407a6" exitCode=60 Oct 29 23:30:04.508684 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.508519 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" event={"ID":"dd068bac-7513-4e03-ab75-5f6777222571","Type":"ContainerDied","Data":"999d37caf6495b9c284c2422f743c6678d4f6d5380160922ad5e8991995407a6"} Oct 29 23:30:04.509174 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.509146 6507 scope.go:122] "RemoveContainer" containerID="999d37caf6495b9c284c2422f743c6678d4f6d5380160922ad5e8991995407a6" Oct 29 23:30:04.509542 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:04.509518 6507 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888610-pk9cq_odh-ai-gateway-infra(dd068bac-7513-4e03-ab75-5f6777222571)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" podUID="dd068bac-7513-4e03-ab75-5f6777222571" Oct 29 23:30:04.511315 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.511285 6507 generic.go:356] "Generic (PLEG): container finished" podID="fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" containerID="2b2054521df19ab97b6c737dc38a94dbd8491db886a8179a97b46ce93c7ce62c" exitCode=60 Oct 29 23:30:04.511403 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.511317 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" event={"ID":"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6","Type":"ContainerDied","Data":"2b2054521df19ab97b6c737dc38a94dbd8491db886a8179a97b46ce93c7ce62c"} Oct 29 23:30:04.511741 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.511724 6507 scope.go:122] "RemoveContainer" containerID="2b2054521df19ab97b6c737dc38a94dbd8491db886a8179a97b46ce93c7ce62c" Oct 29 23:30:04.512162 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:04.512137 6507 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg_odh-ai-gateway-infra(fa54faa3-3bb0-477e-9e21-a44b1d5a92b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" podUID="fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" Oct 29 23:30:04.514323 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.514305 6507 scope.go:122] "RemoveContainer" containerID="170ad77dd5061f7556909d4ac87e47f624541c9e21187b0cc25688e956b4035f" Oct 29 23:30:04.514760 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:04.514733 6507 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz_odh-ai-gateway-infra(3a417439-524a-4ee1-ab88-3f083c09c179)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" podUID="3a417439-524a-4ee1-ab88-3f083c09c179" Oct 29 23:30:04.519698 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.519671 6507 scope.go:122] "RemoveContainer" containerID="3f657a3ace2b9bfdaf770aaab159fe331d503ee4dcd96864bd84178ecb9a996b" Oct 29 23:30:04.536054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.536025 6507 scope.go:122] "RemoveContainer" containerID="880c9a4d3f57dda3b742096f2813f5b5ba716a7592f47e3c9bb883af193f212d" Oct 29 23:30:04.556541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:04.556519 6507 scope.go:122] "RemoveContainer" containerID="cc3a2a3c3ea7c9c95f984e1f2ec4c1440c5d7b1bd485918944358e99c6b7fe5d" Oct 29 23:30:05.353480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.333743 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" Oct 29 23:30:05.515480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.507695 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6173dedb-87b4-4028-9506-34ffadb93263-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6173dedb-87b4-4028-9506-34ffadb93263-plugins-config-volume\") pod \"6173dedb-87b4-4028-9506-34ffadb93263\" (UID: \"6173dedb-87b4-4028-9506-34ffadb93263\") " Oct 29 23:30:05.515480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.508112 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6173dedb-87b4-4028-9506-34ffadb93263-kube-api-access-4dqnt\" (UniqueName: \"kubernetes.io/projected/6173dedb-87b4-4028-9506-34ffadb93263-kube-api-access-4dqnt\") pod \"6173dedb-87b4-4028-9506-34ffadb93263\" (UID: \"6173dedb-87b4-4028-9506-34ffadb93263\") " Oct 29 23:30:05.515480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.509931 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6173dedb-87b4-4028-9506-34ffadb93263-plugins-config-volume" pod "6173dedb-87b4-4028-9506-34ffadb93263" (UID: "6173dedb-87b4-4028-9506-34ffadb93263"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:05.640732 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.635270 6507 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6173dedb-87b4-4028-9506-34ffadb93263-plugins-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:05.640732 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.635446 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6173dedb-87b4-4028-9506-34ffadb93263-kube-api-access-4dqnt" pod "6173dedb-87b4-4028-9506-34ffadb93263" (UID: "6173dedb-87b4-4028-9506-34ffadb93263"). InnerVolumeSpecName "kube-api-access-4dqnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:05.656578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.655376 6507 scope.go:122] "RemoveContainer" containerID="999d37caf6495b9c284c2422f743c6678d4f6d5380160922ad5e8991995407a6" Oct 29 23:30:05.656578 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:05.655867 6507 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888610-pk9cq_odh-ai-gateway-infra(dd068bac-7513-4e03-ab75-5f6777222571)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" podUID="dd068bac-7513-4e03-ab75-5f6777222571" Oct 29 23:30:05.733927 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.733882 6507 scope.go:122] "RemoveContainer" containerID="2b2054521df19ab97b6c737dc38a94dbd8491db886a8179a97b46ce93c7ce62c" Oct 29 23:30:05.734797 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:05.734763 6507 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg_odh-ai-gateway-infra(fa54faa3-3bb0-477e-9e21-a44b1d5a92b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" podUID="fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" Oct 29 23:30:05.802959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.802909 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dqnt\" (UniqueName: \"kubernetes.io/projected/6173dedb-87b4-4028-9506-34ffadb93263-kube-api-access-4dqnt\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:05.809488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.806611 6507 generic.go:356] "Generic (PLEG): container finished" podID="6173dedb-87b4-4028-9506-34ffadb93263" containerID="50615357a1b5beac4da23d76a90f5e586e298be3251d9682c006396ef94685ec" exitCode=0 Oct 29 23:30:05.809488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.806778 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" event={"ID":"6173dedb-87b4-4028-9506-34ffadb93263","Type":"ContainerDied","Data":"50615357a1b5beac4da23d76a90f5e586e298be3251d9682c006396ef94685ec"} Oct 29 23:30:05.809488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.806845 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" event={"ID":"6173dedb-87b4-4028-9506-34ffadb93263","Type":"ContainerDied","Data":"244373564e5e8615dada25569891bb2fd3de777ccbbaccc5873d556c1f352054"} Oct 29 23:30:05.809488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.806887 6507 scope.go:122] "RemoveContainer" containerID="50615357a1b5beac4da23d76a90f5e586e298be3251d9682c006396ef94685ec" Oct 29 23:30:05.809488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.807043 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc" Oct 29 23:30:05.826161 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.824640 6507 generic.go:356] "Generic (PLEG): container finished" podID="9c730892-1170-44b8-9fcf-e74a43035722" containerID="d3b14bd874fd59a3a847a785864271dc7273082d9b3253319e41e78f2cbb0f0d" exitCode=0 Oct 29 23:30:05.826161 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.824757 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" event={"ID":"9c730892-1170-44b8-9fcf-e74a43035722","Type":"ContainerDied","Data":"d3b14bd874fd59a3a847a785864271dc7273082d9b3253319e41e78f2cbb0f0d"} Oct 29 23:30:05.830937 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.829362 6507 scope.go:122] "RemoveContainer" containerID="e6087de0b92f77b2cf63b74441aa9bb5b2d30281ad898381ed536548c65d1cff" Oct 29 23:30:05.830937 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:05.829840 6507 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j_odh-ai-gateway-infra(185d4120-7013-49b4-af2d-0ec8aa4a7f14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" podUID="185d4120-7013-49b4-af2d-0ec8aa4a7f14" Oct 29 23:30:05.831539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:05.831518 6507 scope.go:122] "RemoveContainer" containerID="a74780b5a9fb47af836a17734362930ab6f62832cf5544f42ac2857396b0f36e" Oct 29 23:30:05.832478 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:05.832434 6507 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj_odh-ai-gateway-infra(5f0f0d26-ae35-48ff-9ee6-d591f159ffcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" podUID="5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" Oct 29 23:30:06.003152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.002791 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7"] Oct 29 23:30:06.008814 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.008754 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-313d9c-gbqp7"] Oct 29 23:30:06.011158 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.011037 6507 scope.go:122] "RemoveContainer" containerID="50615357a1b5beac4da23d76a90f5e586e298be3251d9682c006396ef94685ec" Oct 29 23:30:06.012534 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:06.012403 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50615357a1b5beac4da23d76a90f5e586e298be3251d9682c006396ef94685ec\": container with ID starting with 50615357a1b5beac4da23d76a90f5e586e298be3251d9682c006396ef94685ec not found: ID does not exist" containerID="50615357a1b5beac4da23d76a90f5e586e298be3251d9682c006396ef94685ec" Oct 29 23:30:06.012534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.012445 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50615357a1b5beac4da23d76a90f5e586e298be3251d9682c006396ef94685ec"} err="failed to get container status \"50615357a1b5beac4da23d76a90f5e586e298be3251d9682c006396ef94685ec\": rpc error: code = NotFound desc = could not find container \"50615357a1b5beac4da23d76a90f5e586e298be3251d9682c006396ef94685ec\": container with ID starting with 50615357a1b5beac4da23d76a90f5e586e298be3251d9682c006396ef94685ec not found: ID does not exist" Oct 29 23:30:06.015242 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.015213 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc"] Oct 29 23:30:06.019979 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.019951 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-313d9c-67549fc7b6-v5mnc"] Oct 29 23:30:06.152488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.151801 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" Oct 29 23:30:06.218847 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.218806 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c730892-1170-44b8-9fcf-e74a43035722-kube-api-access-g2t2r\" (UniqueName: \"kubernetes.io/projected/9c730892-1170-44b8-9fcf-e74a43035722-kube-api-access-g2t2r\") pod \"9c730892-1170-44b8-9fcf-e74a43035722\" (UID: \"9c730892-1170-44b8-9fcf-e74a43035722\") " Oct 29 23:30:06.218998 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.218908 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c730892-1170-44b8-9fcf-e74a43035722-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c730892-1170-44b8-9fcf-e74a43035722-plugins-config-volume\") pod \"9c730892-1170-44b8-9fcf-e74a43035722\" (UID: \"9c730892-1170-44b8-9fcf-e74a43035722\") " Oct 29 23:30:06.219427 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.219347 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c730892-1170-44b8-9fcf-e74a43035722-plugins-config-volume" pod "9c730892-1170-44b8-9fcf-e74a43035722" (UID: "9c730892-1170-44b8-9fcf-e74a43035722"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:06.219549 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.219510 6507 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c730892-1170-44b8-9fcf-e74a43035722-plugins-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:06.221270 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.221243 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c730892-1170-44b8-9fcf-e74a43035722-kube-api-access-g2t2r" pod "9c730892-1170-44b8-9fcf-e74a43035722" (UID: "9c730892-1170-44b8-9fcf-e74a43035722"). InnerVolumeSpecName "kube-api-access-g2t2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:06.320573 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.320433 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2t2r\" (UniqueName: \"kubernetes.io/projected/9c730892-1170-44b8-9fcf-e74a43035722-kube-api-access-g2t2r\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:06.754488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.754439 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a10cdb-a7bc-423f-97f3-deb0574c3540" path="/var/lib/kubelet/pods/28a10cdb-a7bc-423f-97f3-deb0574c3540/volumes" Oct 29 23:30:06.754942 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.754928 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6173dedb-87b4-4028-9506-34ffadb93263" path="/var/lib/kubelet/pods/6173dedb-87b4-4028-9506-34ffadb93263/volumes" Oct 29 23:30:06.839395 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.839362 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" Oct 29 23:30:06.839395 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.839383 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7" event={"ID":"9c730892-1170-44b8-9fcf-e74a43035722","Type":"ContainerDied","Data":"9f4f1fa5b9587d8aec3b3f9aac6cdfb8ade2c3a882c7b16c0f8c4f8380bc2979"} Oct 29 23:30:06.839673 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.839437 6507 scope.go:122] "RemoveContainer" containerID="d3b14bd874fd59a3a847a785864271dc7273082d9b3253319e41e78f2cbb0f0d" Oct 29 23:30:06.887173 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.887141 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7"] Oct 29 23:30:06.895965 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:06.895933 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-313d9c-5f544fff59-h6tp7"] Oct 29 23:30:08.543180 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:08.543140 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" podUID="10f9be75-9161-4cd4-9b75-8428406eed15" containerName="payload-processing" containerID="cri-o://3cc034b607740335b7b1c3dbdbdb80110f3640cb10d1bc98d4e6bd3a6a41bda3" gracePeriod=25 Oct 29 23:30:08.577847 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:08.577806 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" podUID="50c831dd-76a8-4e42-90b4-19847ae771e4" containerName="payload-pre-processing" containerID="cri-o://0ebd4baad9857e2bd2e0580783ff3ee5b2953d22c1239a124f1ff8b08ac10097" gracePeriod=25 Oct 29 23:30:08.756213 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:08.756181 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c730892-1170-44b8-9fcf-e74a43035722" path="/var/lib/kubelet/pods/9c730892-1170-44b8-9fcf-e74a43035722/volumes" Oct 29 23:30:09.515683 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:09.513666 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" podUID="10f9be75-9161-4cd4-9b75-8428406eed15" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:30:09.515683 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:09.514422 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" podUID="50c831dd-76a8-4e42-90b4-19847ae771e4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:30:11.577280 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.577222 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" containerID="cri-o://e51b6ce1ed688be641124d0f999c15714d4b1c652424539905596a93cd72dbd1" gracePeriod=45 Oct 29 23:30:11.868000 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.867972 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:30:11.897445 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.897412 6507 generic.go:356] "Generic (PLEG): container finished" podID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerID="e51b6ce1ed688be641124d0f999c15714d4b1c652424539905596a93cd72dbd1" exitCode=0 Oct 29 23:30:11.897643 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.897490 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" Oct 29 23:30:11.897643 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.897509 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" event={"ID":"46fb6a01-f848-4ae7-86dc-26e45162b41c","Type":"ContainerDied","Data":"e51b6ce1ed688be641124d0f999c15714d4b1c652424539905596a93cd72dbd1"} Oct 29 23:30:11.897643 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.897548 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk" event={"ID":"46fb6a01-f848-4ae7-86dc-26e45162b41c","Type":"ContainerDied","Data":"ce36bdd9f3dcf051d951bd7ddd0eae8dde7495c33e720925f8b6c9ab35b7df85"} Oct 29 23:30:11.897643 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.897563 6507 scope.go:122] "RemoveContainer" containerID="e51b6ce1ed688be641124d0f999c15714d4b1c652424539905596a93cd72dbd1" Oct 29 23:30:11.911098 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.911066 6507 scope.go:122] "RemoveContainer" containerID="e51b6ce1ed688be641124d0f999c15714d4b1c652424539905596a93cd72dbd1" Oct 29 23:30:11.911587 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:11.911558 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e51b6ce1ed688be641124d0f999c15714d4b1c652424539905596a93cd72dbd1\": container with ID starting with e51b6ce1ed688be641124d0f999c15714d4b1c652424539905596a93cd72dbd1 not found: ID does not exist" containerID="e51b6ce1ed688be641124d0f999c15714d4b1c652424539905596a93cd72dbd1" Oct 29 23:30:11.911748 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.911720 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e51b6ce1ed688be641124d0f999c15714d4b1c652424539905596a93cd72dbd1"} err="failed to get container status \"e51b6ce1ed688be641124d0f999c15714d4b1c652424539905596a93cd72dbd1\": rpc error: code = NotFound desc = could not find container \"e51b6ce1ed688be641124d0f999c15714d4b1c652424539905596a93cd72dbd1\": container with ID starting with e51b6ce1ed688be641124d0f999c15714d4b1c652424539905596a93cd72dbd1 not found: ID does not exist" Oct 29 23:30:11.932077 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.932049 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46fb6a01-f848-4ae7-86dc-26e45162b41c-tls-certs\" (UniqueName: \"kubernetes.io/secret/46fb6a01-f848-4ae7-86dc-26e45162b41c-tls-certs\") pod \"46fb6a01-f848-4ae7-86dc-26e45162b41c\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " Oct 29 23:30:11.932246 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.932170 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-model-cache\") pod \"46fb6a01-f848-4ae7-86dc-26e45162b41c\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " Oct 29 23:30:11.932246 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.932193 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46fb6a01-f848-4ae7-86dc-26e45162b41c-kube-api-access-ncsjr\" (UniqueName: \"kubernetes.io/projected/46fb6a01-f848-4ae7-86dc-26e45162b41c-kube-api-access-ncsjr\") pod \"46fb6a01-f848-4ae7-86dc-26e45162b41c\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " Oct 29 23:30:11.932246 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.932214 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-home\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-home\") pod \"46fb6a01-f848-4ae7-86dc-26e45162b41c\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " Oct 29 23:30:11.932246 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.932235 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-dshm\") pod \"46fb6a01-f848-4ae7-86dc-26e45162b41c\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " Oct 29 23:30:11.932471 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.932443 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-tmp-dir\") pod \"46fb6a01-f848-4ae7-86dc-26e45162b41c\" (UID: \"46fb6a01-f848-4ae7-86dc-26e45162b41c\") " Oct 29 23:30:11.932555 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.932450 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-model-cache" pod "46fb6a01-f848-4ae7-86dc-26e45162b41c" (UID: "46fb6a01-f848-4ae7-86dc-26e45162b41c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:11.932926 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.932898 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-tmp-dir" pod "46fb6a01-f848-4ae7-86dc-26e45162b41c" (UID: "46fb6a01-f848-4ae7-86dc-26e45162b41c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:11.933012 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.932909 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-home" pod "46fb6a01-f848-4ae7-86dc-26e45162b41c" (UID: "46fb6a01-f848-4ae7-86dc-26e45162b41c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:11.933208 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.933183 6507 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-tmp-dir\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:11.933208 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.933207 6507 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-model-cache\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:11.933383 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.933216 6507 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-home\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:11.935074 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.935052 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46fb6a01-f848-4ae7-86dc-26e45162b41c-tls-certs" pod "46fb6a01-f848-4ae7-86dc-26e45162b41c" (UID: "46fb6a01-f848-4ae7-86dc-26e45162b41c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:30:11.935397 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.935369 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-dshm" pod "46fb6a01-f848-4ae7-86dc-26e45162b41c" (UID: "46fb6a01-f848-4ae7-86dc-26e45162b41c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:11.935519 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:11.935422 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fb6a01-f848-4ae7-86dc-26e45162b41c-kube-api-access-ncsjr" pod "46fb6a01-f848-4ae7-86dc-26e45162b41c" (UID: "46fb6a01-f848-4ae7-86dc-26e45162b41c"). InnerVolumeSpecName "kube-api-access-ncsjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:12.034851 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.034818 6507 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46fb6a01-f848-4ae7-86dc-26e45162b41c-tls-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:12.034851 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.034848 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncsjr\" (UniqueName: \"kubernetes.io/projected/46fb6a01-f848-4ae7-86dc-26e45162b41c-kube-api-access-ncsjr\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:12.034851 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.034858 6507 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46fb6a01-f848-4ae7-86dc-26e45162b41c-dshm\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:12.253588 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.253541 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k"] Oct 29 23:30:12.260514 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.260372 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk"] Oct 29 23:30:12.265978 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.265936 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-032fa8/distinct-2-w4-032fa8-kserve-85b4c49474-25lmk"] Oct 29 23:30:12.415664 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.415624 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" containerID="cri-o://e39d9fe266f398488311b1d5831b69fa8b47da0bac68ce2f76cd06d04b824ebb" gracePeriod=45 Oct 29 23:30:12.531575 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.531539 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": dial tcp 10.133.0.51:8000: connect: connection refused" Oct 29 23:30:12.709481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.692069 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:30:12.798541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.791627 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-model-cache\") pod \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " Oct 29 23:30:12.798541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.791791 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-home\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-home\") pod \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " Oct 29 23:30:12.798541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.791924 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-tmp-dir\") pod \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " Oct 29 23:30:12.798541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.791978 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-dshm\") pod \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " Oct 29 23:30:12.798541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.792051 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/045c37c8-57a0-4ed1-87a6-89f73eb079d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/045c37c8-57a0-4ed1-87a6-89f73eb079d2-tls-certs\") pod \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " Oct 29 23:30:12.798541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.792088 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045c37c8-57a0-4ed1-87a6-89f73eb079d2-kube-api-access-x6gm8\" (UniqueName: \"kubernetes.io/projected/045c37c8-57a0-4ed1-87a6-89f73eb079d2-kube-api-access-x6gm8\") pod \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\" (UID: \"045c37c8-57a0-4ed1-87a6-89f73eb079d2\") " Oct 29 23:30:12.819500 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.816752 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-home" pod "045c37c8-57a0-4ed1-87a6-89f73eb079d2" (UID: "045c37c8-57a0-4ed1-87a6-89f73eb079d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:12.819500 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.817221 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-tmp-dir" pod "045c37c8-57a0-4ed1-87a6-89f73eb079d2" (UID: "045c37c8-57a0-4ed1-87a6-89f73eb079d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:12.856973 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.845822 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-model-cache" pod "045c37c8-57a0-4ed1-87a6-89f73eb079d2" (UID: "045c37c8-57a0-4ed1-87a6-89f73eb079d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:12.856973 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.855896 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" path="/var/lib/kubelet/pods/46fb6a01-f848-4ae7-86dc-26e45162b41c/volumes" Oct 29 23:30:12.901506 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.894067 6507 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-home\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:12.901506 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.894105 6507 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-tmp-dir\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:12.901506 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.894120 6507 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-model-cache\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:12.930492 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.930064 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045c37c8-57a0-4ed1-87a6-89f73eb079d2-kube-api-access-x6gm8" pod "045c37c8-57a0-4ed1-87a6-89f73eb079d2" (UID: "045c37c8-57a0-4ed1-87a6-89f73eb079d2"). InnerVolumeSpecName "kube-api-access-x6gm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:12.975566 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:12.974018 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-dshm" pod "045c37c8-57a0-4ed1-87a6-89f73eb079d2" (UID: "045c37c8-57a0-4ed1-87a6-89f73eb079d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:13.001606 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.000127 6507 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/045c37c8-57a0-4ed1-87a6-89f73eb079d2-dshm\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:13.001606 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.000168 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6gm8\" (UniqueName: \"kubernetes.io/projected/045c37c8-57a0-4ed1-87a6-89f73eb079d2-kube-api-access-x6gm8\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:13.004617 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.004582 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/045c37c8-57a0-4ed1-87a6-89f73eb079d2-tls-certs" pod "045c37c8-57a0-4ed1-87a6-89f73eb079d2" (UID: "045c37c8-57a0-4ed1-87a6-89f73eb079d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:30:13.033567 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.033536 6507 generic.go:356] "Generic (PLEG): container finished" podID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerID="e39d9fe266f398488311b1d5831b69fa8b47da0bac68ce2f76cd06d04b824ebb" exitCode=0 Oct 29 23:30:13.033793 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.033770 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" event={"ID":"045c37c8-57a0-4ed1-87a6-89f73eb079d2","Type":"ContainerDied","Data":"e39d9fe266f398488311b1d5831b69fa8b47da0bac68ce2f76cd06d04b824ebb"} Oct 29 23:30:13.034031 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.033813 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" event={"ID":"045c37c8-57a0-4ed1-87a6-89f73eb079d2","Type":"ContainerDied","Data":"0fde8e68d19a5434e260c8dc283a533b771af5c3889ffdae8f2012e2e98d3881"} Oct 29 23:30:13.034031 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.033838 6507 scope.go:122] "RemoveContainer" containerID="e39d9fe266f398488311b1d5831b69fa8b47da0bac68ce2f76cd06d04b824ebb" Oct 29 23:30:13.034520 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.034496 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf" Oct 29 23:30:13.053761 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.053733 6507 scope.go:122] "RemoveContainer" containerID="e39d9fe266f398488311b1d5831b69fa8b47da0bac68ce2f76cd06d04b824ebb" Oct 29 23:30:13.054112 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:13.054089 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39d9fe266f398488311b1d5831b69fa8b47da0bac68ce2f76cd06d04b824ebb\": container with ID starting with e39d9fe266f398488311b1d5831b69fa8b47da0bac68ce2f76cd06d04b824ebb not found: ID does not exist" containerID="e39d9fe266f398488311b1d5831b69fa8b47da0bac68ce2f76cd06d04b824ebb" Oct 29 23:30:13.054194 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.054118 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39d9fe266f398488311b1d5831b69fa8b47da0bac68ce2f76cd06d04b824ebb"} err="failed to get container status \"e39d9fe266f398488311b1d5831b69fa8b47da0bac68ce2f76cd06d04b824ebb\": rpc error: code = NotFound desc = could not find container \"e39d9fe266f398488311b1d5831b69fa8b47da0bac68ce2f76cd06d04b824ebb\": container with ID starting with e39d9fe266f398488311b1d5831b69fa8b47da0bac68ce2f76cd06d04b824ebb not found: ID does not exist" Oct 29 23:30:13.089022 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.088988 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf"] Oct 29 23:30:13.093132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.093104 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-032fa8/premium-w4-032fa8-kserve-c674d5b7d-sv4kf"] Oct 29 23:30:13.101931 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.101907 6507 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/045c37c8-57a0-4ed1-87a6-89f73eb079d2-tls-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:13.492665 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.492626 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn"] Oct 29 23:30:13.492947 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.492916 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" podUID="c272aaa1-73f3-4455-abe8-c61ca1047de3" containerName="istio-proxy" containerID="cri-o://291a70375f0f73d7788e008e0ab608bc39055e9765bda3c96d26c656db9b4736" gracePeriod=30 Oct 29 23:30:13.501910 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.501879 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc"] Oct 29 23:30:13.502120 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.502094 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" podUID="914b9029-3376-4fad-addf-ce1b73391b5e" containerName="istio-proxy" containerID="cri-o://c1c31919fa712acb0dbfc994634efe6f143fbe9caaba34b744c7a47db4519fb0" gracePeriod=30 Oct 29 23:30:13.547421 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.547377 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf"] Oct 29 23:30:13.547819 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.547772 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" podUID="25fce51a-8d4e-4ac0-87e2-68457f9f10df" containerName="istio-proxy" containerID="cri-o://b8b669694b4318f57f1119742bfbeea79fc5e59b6a08e282100fb62a4b320c56" gracePeriod=30 Oct 29 23:30:13.573128 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.573086 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k"] Oct 29 23:30:13.579628 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.579557 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv"] Oct 29 23:30:13.579977 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.579937 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" podUID="089367aa-c3e8-4f67-af00-697e5d7555bd" containerName="istio-proxy" containerID="cri-o://02c2a6d62d3edfe579b09c3bdc20737570e9c65ca27f5d270f09ec6c3f4bb0de" gracePeriod=30 Oct 29 23:30:13.584970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.584932 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr"] Oct 29 23:30:13.756656 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.749161 6507 scope.go:122] "RemoveContainer" containerID="2b2054521df19ab97b6c737dc38a94dbd8491db886a8179a97b46ce93c7ce62c" Oct 29 23:30:13.780785 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:13.780722 6507 scope.go:122] "RemoveContainer" containerID="170ad77dd5061f7556909d4ac87e47f624541c9e21187b0cc25688e956b4035f" Oct 29 23:30:14.102969 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:14.102890 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:30:14.746682 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:14.746647 6507 scope.go:122] "RemoveContainer" containerID="e6087de0b92f77b2cf63b74441aa9bb5b2d30281ad898381ed536548c65d1cff" Oct 29 23:30:14.756355 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:14.756325 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" path="/var/lib/kubelet/pods/045c37c8-57a0-4ed1-87a6-89f73eb079d2/volumes" Oct 29 23:30:15.148267 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:15.148236 6507 generic.go:356] "Generic (PLEG): container finished" podID="fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" containerID="f44e4ce4d8cf96c5c1ea4b5369b66f3921f921c71c6201a87d7060394046fcfe" exitCode=60 Oct 29 23:30:15.148840 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:15.148811 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" event={"ID":"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6","Type":"ContainerDied","Data":"f44e4ce4d8cf96c5c1ea4b5369b66f3921f921c71c6201a87d7060394046fcfe"} Oct 29 23:30:15.149033 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:15.149014 6507 scope.go:122] "RemoveContainer" containerID="2b2054521df19ab97b6c737dc38a94dbd8491db886a8179a97b46ce93c7ce62c" Oct 29 23:30:15.149738 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:15.149721 6507 scope.go:122] "RemoveContainer" containerID="f44e4ce4d8cf96c5c1ea4b5369b66f3921f921c71c6201a87d7060394046fcfe" Oct 29 23:30:15.151090 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:15.151066 6507 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg_odh-ai-gateway-infra(fa54faa3-3bb0-477e-9e21-a44b1d5a92b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" podUID="fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" Oct 29 23:30:15.153782 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:15.153758 6507 generic.go:356] "Generic (PLEG): container finished" podID="3a417439-524a-4ee1-ab88-3f083c09c179" containerID="1e35a386e6ecc1f0884652ab409819bae6439a2acfcb3df63c2f7ce482ca7738" exitCode=60 Oct 29 23:30:15.154041 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:15.154024 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" event={"ID":"3a417439-524a-4ee1-ab88-3f083c09c179","Type":"ContainerDied","Data":"1e35a386e6ecc1f0884652ab409819bae6439a2acfcb3df63c2f7ce482ca7738"} Oct 29 23:30:15.154625 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:15.154607 6507 scope.go:122] "RemoveContainer" containerID="1e35a386e6ecc1f0884652ab409819bae6439a2acfcb3df63c2f7ce482ca7738" Oct 29 23:30:15.155117 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:15.155098 6507 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz_odh-ai-gateway-infra(3a417439-524a-4ee1-ab88-3f083c09c179)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" podUID="3a417439-524a-4ee1-ab88-3f083c09c179" Oct 29 23:30:15.167795 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:15.167772 6507 scope.go:122] "RemoveContainer" containerID="170ad77dd5061f7556909d4ac87e47f624541c9e21187b0cc25688e956b4035f" Oct 29 23:30:15.243499 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:15.242592 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:30:15.747481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:15.747373 6507 scope.go:122] "RemoveContainer" containerID="999d37caf6495b9c284c2422f743c6678d4f6d5380160922ad5e8991995407a6" Oct 29 23:30:15.798282 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:15.798243 6507 scope.go:122] "RemoveContainer" containerID="a74780b5a9fb47af836a17734362930ab6f62832cf5544f42ac2857396b0f36e" Oct 29 23:30:16.231484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:16.230505 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg"] Oct 29 23:30:16.418791 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:16.417856 6507 generic.go:356] "Generic (PLEG): container finished" podID="185d4120-7013-49b4-af2d-0ec8aa4a7f14" containerID="a317d9c2dd6506953250e8aef084e1f0e907ca6d12f22a08d856c0571202e7bf" exitCode=60 Oct 29 23:30:16.418791 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:16.417983 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" event={"ID":"185d4120-7013-49b4-af2d-0ec8aa4a7f14","Type":"ContainerDied","Data":"a317d9c2dd6506953250e8aef084e1f0e907ca6d12f22a08d856c0571202e7bf"} Oct 29 23:30:16.418791 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:16.418044 6507 scope.go:122] "RemoveContainer" containerID="e6087de0b92f77b2cf63b74441aa9bb5b2d30281ad898381ed536548c65d1cff" Oct 29 23:30:16.421301 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:16.419367 6507 scope.go:122] "RemoveContainer" containerID="a317d9c2dd6506953250e8aef084e1f0e907ca6d12f22a08d856c0571202e7bf" Oct 29 23:30:16.423068 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:16.423034 6507 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j_odh-ai-gateway-infra(185d4120-7013-49b4-af2d-0ec8aa4a7f14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" podUID="185d4120-7013-49b4-af2d-0ec8aa4a7f14" Oct 29 23:30:16.519203 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:16.519165 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz"] Oct 29 23:30:16.778735 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:16.769705 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:30:16.778735 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:16.769779 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" podUID="c272aaa1-73f3-4455-abe8-c61ca1047de3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:30:17.880232 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:17.873379 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 29 23:30:17.935834 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:17.930699 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 29 23:30:18.219032 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.217847 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j"] Oct 29 23:30:18.250483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.245259 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" Oct 29 23:30:18.301113 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.301055 6507 generic.go:356] "Generic (PLEG): container finished" podID="dd068bac-7513-4e03-ab75-5f6777222571" containerID="d88cd9c438ac4d8d17624edb92ccbfd5109bfadf14c146045f6673cf2cb7ac39" exitCode=60 Oct 29 23:30:18.301469 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.301426 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" event={"ID":"dd068bac-7513-4e03-ab75-5f6777222571","Type":"ContainerDied","Data":"d88cd9c438ac4d8d17624edb92ccbfd5109bfadf14c146045f6673cf2cb7ac39"} Oct 29 23:30:18.301747 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.301726 6507 scope.go:122] "RemoveContainer" containerID="999d37caf6495b9c284c2422f743c6678d4f6d5380160922ad5e8991995407a6" Oct 29 23:30:18.303109 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.303088 6507 scope.go:122] "RemoveContainer" containerID="d88cd9c438ac4d8d17624edb92ccbfd5109bfadf14c146045f6673cf2cb7ac39" Oct 29 23:30:18.315131 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:18.313744 6507 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888610-pk9cq_odh-ai-gateway-infra(dd068bac-7513-4e03-ab75-5f6777222571)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" podUID="dd068bac-7513-4e03-ab75-5f6777222571" Oct 29 23:30:18.315131 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.305212 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" Oct 29 23:30:18.315131 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.306920 6507 patch_prober.go:29] interesting pod/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:30:18.315131 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.313966 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" podUID="25fce51a-8d4e-4ac0-87e2-68457f9f10df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:30:18.315131 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.308792 6507 generic.go:356] "Generic (PLEG): container finished" podID="5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" containerID="e3dc67a93e7cd4b4d012e4ed4b7ee878be8e41e333d219ad2deb993e5ed18ec5" exitCode=60 Oct 29 23:30:18.315131 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.305128 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg" event={"ID":"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6","Type":"ContainerDied","Data":"651c81c98d7bf3b244c8fe70f35da83b4a2a3c33cb7d5afe69e10f70e7a93bc4"} Oct 29 23:30:18.315131 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.314190 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" event={"ID":"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc","Type":"ContainerDied","Data":"e3dc67a93e7cd4b4d012e4ed4b7ee878be8e41e333d219ad2deb993e5ed18ec5"} Oct 29 23:30:18.315131 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.314665 6507 scope.go:122] "RemoveContainer" containerID="e3dc67a93e7cd4b4d012e4ed4b7ee878be8e41e333d219ad2deb993e5ed18ec5" Oct 29 23:30:18.315131 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:18.315027 6507 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj_odh-ai-gateway-infra(5f0f0d26-ae35-48ff-9ee6-d591f159ffcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" podUID="5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" Oct 29 23:30:18.318792 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.318766 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" Oct 29 23:30:18.337732 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.337222 6507 scope.go:122] "RemoveContainer" containerID="f44e4ce4d8cf96c5c1ea4b5369b66f3921f921c71c6201a87d7060394046fcfe" Oct 29 23:30:18.404565 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.402595 6507 scope.go:122] "RemoveContainer" containerID="a74780b5a9fb47af836a17734362930ab6f62832cf5544f42ac2857396b0f36e" Oct 29 23:30:18.419811 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.419638 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2"] Oct 29 23:30:18.464788 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.461860 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a417439-524a-4ee1-ab88-3f083c09c179-maas-api-service-ca" pod "3a417439-524a-4ee1-ab88-3f083c09c179" (UID: "3a417439-524a-4ee1-ab88-3f083c09c179"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:18.464788 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.461442 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a417439-524a-4ee1-ab88-3f083c09c179-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a417439-524a-4ee1-ab88-3f083c09c179-maas-api-service-ca\") pod \"3a417439-524a-4ee1-ab88-3f083c09c179\" (UID: \"3a417439-524a-4ee1-ab88-3f083c09c179\") " Oct 29 23:30:18.464788 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.461985 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa54faa3-3bb0-477e-9e21-a44b1d5a92b6-kube-api-access-mdw7h\" (UniqueName: \"kubernetes.io/projected/fa54faa3-3bb0-477e-9e21-a44b1d5a92b6-kube-api-access-mdw7h\") pod \"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6\" (UID: \"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6\") " Oct 29 23:30:18.464788 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.462507 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa54faa3-3bb0-477e-9e21-a44b1d5a92b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa54faa3-3bb0-477e-9e21-a44b1d5a92b6-maas-api-service-ca\") pod \"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6\" (UID: \"fa54faa3-3bb0-477e-9e21-a44b1d5a92b6\") " Oct 29 23:30:18.464788 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.462918 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa54faa3-3bb0-477e-9e21-a44b1d5a92b6-maas-api-service-ca" pod "fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" (UID: "fa54faa3-3bb0-477e-9e21-a44b1d5a92b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:18.464788 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.463086 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a417439-524a-4ee1-ab88-3f083c09c179-kube-api-access-dl7gc\" (UniqueName: \"kubernetes.io/projected/3a417439-524a-4ee1-ab88-3f083c09c179-kube-api-access-dl7gc\") pod \"3a417439-524a-4ee1-ab88-3f083c09c179\" (UID: \"3a417439-524a-4ee1-ab88-3f083c09c179\") " Oct 29 23:30:18.464788 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.464207 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa54faa3-3bb0-477e-9e21-a44b1d5a92b6-maas-api-service-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:18.464788 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.464230 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a417439-524a-4ee1-ab88-3f083c09c179-maas-api-service-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:18.473207 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.473174 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2"] Oct 29 23:30:18.476488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.476403 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a417439-524a-4ee1-ab88-3f083c09c179-kube-api-access-dl7gc" pod "3a417439-524a-4ee1-ab88-3f083c09c179" (UID: "3a417439-524a-4ee1-ab88-3f083c09c179"). InnerVolumeSpecName "kube-api-access-dl7gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:18.477021 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.476847 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa54faa3-3bb0-477e-9e21-a44b1d5a92b6-kube-api-access-mdw7h" pod "fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" (UID: "fa54faa3-3bb0-477e-9e21-a44b1d5a92b6"). InnerVolumeSpecName "kube-api-access-mdw7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:18.566035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.565578 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl7gc\" (UniqueName: \"kubernetes.io/projected/3a417439-524a-4ee1-ab88-3f083c09c179-kube-api-access-dl7gc\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:18.566035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.565603 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdw7h\" (UniqueName: \"kubernetes.io/projected/fa54faa3-3bb0-477e-9e21-a44b1d5a92b6-kube-api-access-mdw7h\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:18.827312 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.820783 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" Oct 29 23:30:18.865927 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.864291 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg"] Oct 29 23:30:18.870520 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.870423 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9280e4-29888610-4s8kg"] Oct 29 23:30:18.888810 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.888759 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:30:18.895946 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.895914 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185d4120-7013-49b4-af2d-0ec8aa4a7f14-kube-api-access-7n28g\" (UniqueName: \"kubernetes.io/projected/185d4120-7013-49b4-af2d-0ec8aa4a7f14-kube-api-access-7n28g\") pod \"185d4120-7013-49b4-af2d-0ec8aa4a7f14\" (UID: \"185d4120-7013-49b4-af2d-0ec8aa4a7f14\") " Oct 29 23:30:18.896220 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.896202 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-workload-certs\") pod \"914b9029-3376-4fad-addf-ce1b73391b5e\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " Oct 29 23:30:18.896496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.896344 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-credential-socket\") pod \"914b9029-3376-4fad-addf-ce1b73391b5e\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " Oct 29 23:30:18.896873 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.896841 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-istio-data\") pod \"914b9029-3376-4fad-addf-ce1b73391b5e\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " Oct 29 23:30:18.897039 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.897009 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914b9029-3376-4fad-addf-ce1b73391b5e-istio-token\" (UniqueName: \"kubernetes.io/projected/914b9029-3376-4fad-addf-ce1b73391b5e-istio-token\") pod \"914b9029-3376-4fad-addf-ce1b73391b5e\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " Oct 29 23:30:18.897206 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.897191 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-workload-socket\") pod \"914b9029-3376-4fad-addf-ce1b73391b5e\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " Oct 29 23:30:18.897342 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.897314 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/185d4120-7013-49b4-af2d-0ec8aa4a7f14-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185d4120-7013-49b4-af2d-0ec8aa4a7f14-maas-api-service-ca\") pod \"185d4120-7013-49b4-af2d-0ec8aa4a7f14\" (UID: \"185d4120-7013-49b4-af2d-0ec8aa4a7f14\") " Oct 29 23:30:18.897495 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.897480 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914b9029-3376-4fad-addf-ce1b73391b5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914b9029-3376-4fad-addf-ce1b73391b5e-istiod-ca-cert\") pod \"914b9029-3376-4fad-addf-ce1b73391b5e\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " Oct 29 23:30:18.897711 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.897682 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/914b9029-3376-4fad-addf-ce1b73391b5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914b9029-3376-4fad-addf-ce1b73391b5e-istio-podinfo\") pod \"914b9029-3376-4fad-addf-ce1b73391b5e\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " Oct 29 23:30:18.897892 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.897877 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914b9029-3376-4fad-addf-ce1b73391b5e-kube-api-access-tgwrd\" (UniqueName: \"kubernetes.io/projected/914b9029-3376-4fad-addf-ce1b73391b5e-kube-api-access-tgwrd\") pod \"914b9029-3376-4fad-addf-ce1b73391b5e\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " Oct 29 23:30:18.898046 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.898032 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-istio-envoy\") pod \"914b9029-3376-4fad-addf-ce1b73391b5e\" (UID: \"914b9029-3376-4fad-addf-ce1b73391b5e\") " Oct 29 23:30:18.899001 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.898627 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185d4120-7013-49b4-af2d-0ec8aa4a7f14-kube-api-access-7n28g" pod "185d4120-7013-49b4-af2d-0ec8aa4a7f14" (UID: "185d4120-7013-49b4-af2d-0ec8aa4a7f14"). InnerVolumeSpecName "kube-api-access-7n28g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:18.899001 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.898805 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-workload-socket" pod "914b9029-3376-4fad-addf-ce1b73391b5e" (UID: "914b9029-3376-4fad-addf-ce1b73391b5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:18.899001 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.898994 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-workload-certs" pod "914b9029-3376-4fad-addf-ce1b73391b5e" (UID: "914b9029-3376-4fad-addf-ce1b73391b5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:18.899248 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.899143 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-credential-socket" pod "914b9029-3376-4fad-addf-ce1b73391b5e" (UID: "914b9029-3376-4fad-addf-ce1b73391b5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:18.899357 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.899293 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-istio-data" pod "914b9029-3376-4fad-addf-ce1b73391b5e" (UID: "914b9029-3376-4fad-addf-ce1b73391b5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:18.899953 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.899918 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:18.900084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.900070 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:18.900222 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.900208 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n28g\" (UniqueName: \"kubernetes.io/projected/185d4120-7013-49b4-af2d-0ec8aa4a7f14-kube-api-access-7n28g\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:18.900354 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.900341 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:18.900483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.900468 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:18.904116 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.902797 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914b9029-3376-4fad-addf-ce1b73391b5e-istio-token" pod "914b9029-3376-4fad-addf-ce1b73391b5e" (UID: "914b9029-3376-4fad-addf-ce1b73391b5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:18.904116 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.902877 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-istio-envoy" pod "914b9029-3376-4fad-addf-ce1b73391b5e" (UID: "914b9029-3376-4fad-addf-ce1b73391b5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:18.904116 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.903195 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185d4120-7013-49b4-af2d-0ec8aa4a7f14-maas-api-service-ca" pod "185d4120-7013-49b4-af2d-0ec8aa4a7f14" (UID: "185d4120-7013-49b4-af2d-0ec8aa4a7f14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:18.904116 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.903722 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914b9029-3376-4fad-addf-ce1b73391b5e-istiod-ca-cert" pod "914b9029-3376-4fad-addf-ce1b73391b5e" (UID: "914b9029-3376-4fad-addf-ce1b73391b5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:18.905773 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.905747 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/914b9029-3376-4fad-addf-ce1b73391b5e-istio-podinfo" pod "914b9029-3376-4fad-addf-ce1b73391b5e" (UID: "914b9029-3376-4fad-addf-ce1b73391b5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:30:18.907096 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:18.907070 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914b9029-3376-4fad-addf-ce1b73391b5e-kube-api-access-tgwrd" pod "914b9029-3376-4fad-addf-ce1b73391b5e" (UID: "914b9029-3376-4fad-addf-ce1b73391b5e"). InnerVolumeSpecName "kube-api-access-tgwrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:19.002471 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.002420 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914b9029-3376-4fad-addf-ce1b73391b5e-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.002471 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.002469 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185d4120-7013-49b4-af2d-0ec8aa4a7f14-maas-api-service-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.002665 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.002483 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914b9029-3376-4fad-addf-ce1b73391b5e-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.002665 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.002498 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914b9029-3376-4fad-addf-ce1b73391b5e-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.002665 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.002510 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgwrd\" (UniqueName: \"kubernetes.io/projected/914b9029-3376-4fad-addf-ce1b73391b5e-kube-api-access-tgwrd\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.002665 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.002522 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914b9029-3376-4fad-addf-ce1b73391b5e-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.037192 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.037162 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:30:19.081957 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.081929 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:30:19.084929 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.084891 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:30:19.103320 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.103282 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-envoy\") pod \"c272aaa1-73f3-4455-abe8-c61ca1047de3\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " Oct 29 23:30:19.103530 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.103351 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-podinfo\") pod \"c272aaa1-73f3-4455-abe8-c61ca1047de3\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " Oct 29 23:30:19.103530 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.103420 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c272aaa1-73f3-4455-abe8-c61ca1047de3-kube-api-access-7qb89\" (UniqueName: \"kubernetes.io/projected/c272aaa1-73f3-4455-abe8-c61ca1047de3-kube-api-access-7qb89\") pod \"c272aaa1-73f3-4455-abe8-c61ca1047de3\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " Oct 29 23:30:19.103530 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.103512 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-data\") pod \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " Oct 29 23:30:19.103724 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.103547 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-data\") pod \"c272aaa1-73f3-4455-abe8-c61ca1047de3\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " Oct 29 23:30:19.103724 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.103612 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089367aa-c3e8-4f67-af00-697e5d7555bd-kube-api-access-5bfk6\" (UniqueName: \"kubernetes.io/projected/089367aa-c3e8-4f67-af00-697e5d7555bd-kube-api-access-5bfk6\") pod \"089367aa-c3e8-4f67-af00-697e5d7555bd\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " Oct 29 23:30:19.103724 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.103652 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-podinfo\") pod \"089367aa-c3e8-4f67-af00-697e5d7555bd\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " Oct 29 23:30:19.103724 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.103694 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-data\") pod \"089367aa-c3e8-4f67-af00-697e5d7555bd\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " Oct 29 23:30:19.103909 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.103725 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-workload-socket\") pod \"c272aaa1-73f3-4455-abe8-c61ca1047de3\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " Oct 29 23:30:19.103909 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.103804 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-podinfo\") pod \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " Oct 29 23:30:19.103909 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.103847 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-envoy\") pod \"089367aa-c3e8-4f67-af00-697e5d7555bd\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " Oct 29 23:30:19.104054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.103924 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-workload-certs\") pod \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " Oct 29 23:30:19.104054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.103929 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-data" pod "c272aaa1-73f3-4455-abe8-c61ca1047de3" (UID: "c272aaa1-73f3-4455-abe8-c61ca1047de3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:19.104054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.104001 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-envoy\") pod \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " Oct 29 23:30:19.104054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.104042 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/089367aa-c3e8-4f67-af00-697e5d7555bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/089367aa-c3e8-4f67-af00-697e5d7555bd-istiod-ca-cert\") pod \"089367aa-c3e8-4f67-af00-697e5d7555bd\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " Oct 29 23:30:19.104250 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.104128 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-workload-socket\") pod \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " Oct 29 23:30:19.104250 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.104156 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istiod-ca-cert\") pod \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " Oct 29 23:30:19.104250 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.104159 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-data" pod "25fce51a-8d4e-4ac0-87e2-68457f9f10df" (UID: "25fce51a-8d4e-4ac0-87e2-68457f9f10df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:19.104250 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.104186 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-credential-socket\") pod \"c272aaa1-73f3-4455-abe8-c61ca1047de3\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " Oct 29 23:30:19.104439 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.104251 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25fce51a-8d4e-4ac0-87e2-68457f9f10df-kube-api-access-g7m7m\" (UniqueName: \"kubernetes.io/projected/25fce51a-8d4e-4ac0-87e2-68457f9f10df-kube-api-access-g7m7m\") pod \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " Oct 29 23:30:19.104439 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.104281 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-workload-certs\") pod \"089367aa-c3e8-4f67-af00-697e5d7555bd\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " Oct 29 23:30:19.104439 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.104367 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-workload-socket\") pod \"089367aa-c3e8-4f67-af00-697e5d7555bd\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " Oct 29 23:30:19.104439 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.104402 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-token\" (UniqueName: \"kubernetes.io/projected/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-token\") pod \"c272aaa1-73f3-4455-abe8-c61ca1047de3\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " Oct 29 23:30:19.104439 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.104432 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-token\" (UniqueName: \"kubernetes.io/projected/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-token\") pod \"089367aa-c3e8-4f67-af00-697e5d7555bd\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " Oct 29 23:30:19.104762 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.104494 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-token\" (UniqueName: \"kubernetes.io/projected/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-token\") pod \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " Oct 29 23:30:19.104762 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.104570 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-workload-certs\") pod \"c272aaa1-73f3-4455-abe8-c61ca1047de3\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " Oct 29 23:30:19.104762 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.104607 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c272aaa1-73f3-4455-abe8-c61ca1047de3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c272aaa1-73f3-4455-abe8-c61ca1047de3-istiod-ca-cert\") pod \"c272aaa1-73f3-4455-abe8-c61ca1047de3\" (UID: \"c272aaa1-73f3-4455-abe8-c61ca1047de3\") " Oct 29 23:30:19.104762 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.104654 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-credential-socket\") pod \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\" (UID: \"25fce51a-8d4e-4ac0-87e2-68457f9f10df\") " Oct 29 23:30:19.104762 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.104686 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-credential-socket\") pod \"089367aa-c3e8-4f67-af00-697e5d7555bd\" (UID: \"089367aa-c3e8-4f67-af00-697e5d7555bd\") " Oct 29 23:30:19.105417 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.105393 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-workload-certs" pod "089367aa-c3e8-4f67-af00-697e5d7555bd" (UID: "089367aa-c3e8-4f67-af00-697e5d7555bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:19.105642 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.105549 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-workload-socket" pod "25fce51a-8d4e-4ac0-87e2-68457f9f10df" (UID: "25fce51a-8d4e-4ac0-87e2-68457f9f10df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:19.105921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.105761 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-credential-socket" pod "089367aa-c3e8-4f67-af00-697e5d7555bd" (UID: "089367aa-c3e8-4f67-af00-697e5d7555bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:19.105921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.105778 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-data" pod "089367aa-c3e8-4f67-af00-697e5d7555bd" (UID: "089367aa-c3e8-4f67-af00-697e5d7555bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:19.105921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.105907 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-workload-certs" pod "25fce51a-8d4e-4ac0-87e2-68457f9f10df" (UID: "25fce51a-8d4e-4ac0-87e2-68457f9f10df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:19.106169 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.106137 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istiod-ca-cert" pod "25fce51a-8d4e-4ac0-87e2-68457f9f10df" (UID: "25fce51a-8d4e-4ac0-87e2-68457f9f10df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:19.106975 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.106391 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-podinfo" pod "c272aaa1-73f3-4455-abe8-c61ca1047de3" (UID: "c272aaa1-73f3-4455-abe8-c61ca1047de3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:30:19.106975 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.106483 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-credential-socket" pod "c272aaa1-73f3-4455-abe8-c61ca1047de3" (UID: "c272aaa1-73f3-4455-abe8-c61ca1047de3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:19.106975 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.106599 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-workload-socket" pod "089367aa-c3e8-4f67-af00-697e5d7555bd" (UID: "089367aa-c3e8-4f67-af00-697e5d7555bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:19.106975 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.106713 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-workload-socket" pod "c272aaa1-73f3-4455-abe8-c61ca1047de3" (UID: "c272aaa1-73f3-4455-abe8-c61ca1047de3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:19.108113 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.107808 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-envoy" pod "089367aa-c3e8-4f67-af00-697e5d7555bd" (UID: "089367aa-c3e8-4f67-af00-697e5d7555bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:19.108113 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.108075 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c272aaa1-73f3-4455-abe8-c61ca1047de3-kube-api-access-7qb89" pod "c272aaa1-73f3-4455-abe8-c61ca1047de3" (UID: "c272aaa1-73f3-4455-abe8-c61ca1047de3"). InnerVolumeSpecName "kube-api-access-7qb89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:19.108382 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.108360 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-workload-certs" pod "c272aaa1-73f3-4455-abe8-c61ca1047de3" (UID: "c272aaa1-73f3-4455-abe8-c61ca1047de3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:19.109969 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.109085 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-envoy" pod "25fce51a-8d4e-4ac0-87e2-68457f9f10df" (UID: "25fce51a-8d4e-4ac0-87e2-68457f9f10df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:19.109969 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.109166 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089367aa-c3e8-4f67-af00-697e5d7555bd-kube-api-access-5bfk6" pod "089367aa-c3e8-4f67-af00-697e5d7555bd" (UID: "089367aa-c3e8-4f67-af00-697e5d7555bd"). InnerVolumeSpecName "kube-api-access-5bfk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:19.109969 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.109555 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089367aa-c3e8-4f67-af00-697e5d7555bd-istiod-ca-cert" pod "089367aa-c3e8-4f67-af00-697e5d7555bd" (UID: "089367aa-c3e8-4f67-af00-697e5d7555bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:19.109969 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.109634 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-token" pod "c272aaa1-73f3-4455-abe8-c61ca1047de3" (UID: "c272aaa1-73f3-4455-abe8-c61ca1047de3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:19.109969 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.109756 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-envoy" pod "c272aaa1-73f3-4455-abe8-c61ca1047de3" (UID: "c272aaa1-73f3-4455-abe8-c61ca1047de3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:19.109969 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.105571 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.109969 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.109847 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.109969 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.109868 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-credential-socket" pod "25fce51a-8d4e-4ac0-87e2-68457f9f10df" (UID: "25fce51a-8d4e-4ac0-87e2-68457f9f10df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:19.109969 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.109926 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c272aaa1-73f3-4455-abe8-c61ca1047de3-istiod-ca-cert" pod "c272aaa1-73f3-4455-abe8-c61ca1047de3" (UID: "c272aaa1-73f3-4455-abe8-c61ca1047de3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:19.110521 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.110000 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-podinfo" pod "25fce51a-8d4e-4ac0-87e2-68457f9f10df" (UID: "25fce51a-8d4e-4ac0-87e2-68457f9f10df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:30:19.110521 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.110109 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fce51a-8d4e-4ac0-87e2-68457f9f10df-kube-api-access-g7m7m" pod "25fce51a-8d4e-4ac0-87e2-68457f9f10df" (UID: "25fce51a-8d4e-4ac0-87e2-68457f9f10df"). InnerVolumeSpecName "kube-api-access-g7m7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:19.110644 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.110603 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-podinfo" pod "089367aa-c3e8-4f67-af00-697e5d7555bd" (UID: "089367aa-c3e8-4f67-af00-697e5d7555bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:30:19.111145 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.111119 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-token" pod "089367aa-c3e8-4f67-af00-697e5d7555bd" (UID: "089367aa-c3e8-4f67-af00-697e5d7555bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:19.111359 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.111338 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-token" pod "25fce51a-8d4e-4ac0-87e2-68457f9f10df" (UID: "25fce51a-8d4e-4ac0-87e2-68457f9f10df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211561 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bfk6\" (UniqueName: \"kubernetes.io/projected/089367aa-c3e8-4f67-af00-697e5d7555bd-kube-api-access-5bfk6\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211601 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211616 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211628 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211644 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211657 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211669 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211681 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211693 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/089367aa-c3e8-4f67-af00-697e5d7555bd-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211705 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211718 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211730 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211743 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7m7m\" (UniqueName: \"kubernetes.io/projected/25fce51a-8d4e-4ac0-87e2-68457f9f10df-kube-api-access-g7m7m\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211755 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211768 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211778 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211792 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/089367aa-c3e8-4f67-af00-697e5d7555bd-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211805 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25fce51a-8d4e-4ac0-87e2-68457f9f10df-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211815 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211826 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c272aaa1-73f3-4455-abe8-c61ca1047de3-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211837 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fce51a-8d4e-4ac0-87e2-68457f9f10df-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211849 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/089367aa-c3e8-4f67-af00-697e5d7555bd-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211862 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211875 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c272aaa1-73f3-4455-abe8-c61ca1047de3-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.211970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.211886 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qb89\" (UniqueName: \"kubernetes.io/projected/c272aaa1-73f3-4455-abe8-c61ca1047de3-kube-api-access-7qb89\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.392269 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.381915 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" event={"ID":"3a417439-524a-4ee1-ab88-3f083c09c179","Type":"ContainerDied","Data":"a568c3cf4bfbe834db9a5a3f61d0705395d891b8b4fb0486c21020551c1f2606"} Oct 29 23:30:19.392269 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.381989 6507 scope.go:122] "RemoveContainer" containerID="1e35a386e6ecc1f0884652ab409819bae6439a2acfcb3df63c2f7ce482ca7738" Oct 29 23:30:19.392269 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.382096 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz" Oct 29 23:30:19.416490 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.416304 6507 generic.go:356] "Generic (PLEG): container finished" podID="089367aa-c3e8-4f67-af00-697e5d7555bd" containerID="02c2a6d62d3edfe579b09c3bdc20737570e9c65ca27f5d270f09ec6c3f4bb0de" exitCode=0 Oct 29 23:30:19.419198 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.417946 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" event={"ID":"089367aa-c3e8-4f67-af00-697e5d7555bd","Type":"ContainerDied","Data":"02c2a6d62d3edfe579b09c3bdc20737570e9c65ca27f5d270f09ec6c3f4bb0de"} Oct 29 23:30:19.425694 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.422524 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" event={"ID":"089367aa-c3e8-4f67-af00-697e5d7555bd","Type":"ContainerDied","Data":"8b882730a80fc50a4bfa0ed274b007d29245882d22fa74248396ef377799978a"} Oct 29 23:30:19.425694 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.422692 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv" Oct 29 23:30:19.426634 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.426419 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq"] Oct 29 23:30:19.434483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.427414 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" event={"ID":"185d4120-7013-49b4-af2d-0ec8aa4a7f14","Type":"ContainerDied","Data":"fe9e2fe9f7bc66250981339a56e3624cc6de969891b026bb2a0fa27918348fa0"} Oct 29 23:30:19.434483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.427448 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j" Oct 29 23:30:19.434483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.429285 6507 generic.go:356] "Generic (PLEG): container finished" podID="25fce51a-8d4e-4ac0-87e2-68457f9f10df" containerID="b8b669694b4318f57f1119742bfbeea79fc5e59b6a08e282100fb62a4b320c56" exitCode=0 Oct 29 23:30:19.434483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.429328 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" Oct 29 23:30:19.434483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.429361 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" event={"ID":"25fce51a-8d4e-4ac0-87e2-68457f9f10df","Type":"ContainerDied","Data":"b8b669694b4318f57f1119742bfbeea79fc5e59b6a08e282100fb62a4b320c56"} Oct 29 23:30:19.434483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.429406 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf" event={"ID":"25fce51a-8d4e-4ac0-87e2-68457f9f10df","Type":"ContainerDied","Data":"a10ab5736916bf0df0e4094191275503f4c5d3f8d23579dc97f528aa0aa95213"} Oct 29 23:30:19.434483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.431406 6507 generic.go:356] "Generic (PLEG): container finished" podID="914b9029-3376-4fad-addf-ce1b73391b5e" containerID="c1c31919fa712acb0dbfc994634efe6f143fbe9caaba34b744c7a47db4519fb0" exitCode=0 Oct 29 23:30:19.434483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.431572 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" event={"ID":"914b9029-3376-4fad-addf-ce1b73391b5e","Type":"ContainerDied","Data":"c1c31919fa712acb0dbfc994634efe6f143fbe9caaba34b744c7a47db4519fb0"} Oct 29 23:30:19.434483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.431613 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" event={"ID":"914b9029-3376-4fad-addf-ce1b73391b5e","Type":"ContainerDied","Data":"07556dc31cff91821d323abb1e56833c8d1e0e4f3eaee538c4682f53706ff01d"} Oct 29 23:30:19.434483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.431686 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc" Oct 29 23:30:19.436660 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.436631 6507 scope.go:122] "RemoveContainer" containerID="02c2a6d62d3edfe579b09c3bdc20737570e9c65ca27f5d270f09ec6c3f4bb0de" Oct 29 23:30:19.440345 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.440311 6507 generic.go:356] "Generic (PLEG): container finished" podID="c272aaa1-73f3-4455-abe8-c61ca1047de3" containerID="291a70375f0f73d7788e008e0ab608bc39055e9765bda3c96d26c656db9b4736" exitCode=0 Oct 29 23:30:19.440480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.440431 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" event={"ID":"c272aaa1-73f3-4455-abe8-c61ca1047de3","Type":"ContainerDied","Data":"291a70375f0f73d7788e008e0ab608bc39055e9765bda3c96d26c656db9b4736"} Oct 29 23:30:19.440851 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.440829 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" event={"ID":"c272aaa1-73f3-4455-abe8-c61ca1047de3","Type":"ContainerDied","Data":"e8e1496707e5d84866e0190e3eee6c14ce18a90f2ccdc7705c94e5be8cc6dd7d"} Oct 29 23:30:19.440930 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.440917 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn" Oct 29 23:30:19.455628 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.455602 6507 scope.go:122] "RemoveContainer" containerID="02c2a6d62d3edfe579b09c3bdc20737570e9c65ca27f5d270f09ec6c3f4bb0de" Oct 29 23:30:19.455912 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:19.455891 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c2a6d62d3edfe579b09c3bdc20737570e9c65ca27f5d270f09ec6c3f4bb0de\": container with ID starting with 02c2a6d62d3edfe579b09c3bdc20737570e9c65ca27f5d270f09ec6c3f4bb0de not found: ID does not exist" containerID="02c2a6d62d3edfe579b09c3bdc20737570e9c65ca27f5d270f09ec6c3f4bb0de" Oct 29 23:30:19.456013 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.455920 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c2a6d62d3edfe579b09c3bdc20737570e9c65ca27f5d270f09ec6c3f4bb0de"} err="failed to get container status \"02c2a6d62d3edfe579b09c3bdc20737570e9c65ca27f5d270f09ec6c3f4bb0de\": rpc error: code = NotFound desc = could not find container \"02c2a6d62d3edfe579b09c3bdc20737570e9c65ca27f5d270f09ec6c3f4bb0de\": container with ID starting with 02c2a6d62d3edfe579b09c3bdc20737570e9c65ca27f5d270f09ec6c3f4bb0de not found: ID does not exist" Oct 29 23:30:19.456013 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.455938 6507 scope.go:122] "RemoveContainer" containerID="a317d9c2dd6506953250e8aef084e1f0e907ca6d12f22a08d856c0571202e7bf" Oct 29 23:30:19.466666 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.466622 6507 scope.go:122] "RemoveContainer" containerID="b8b669694b4318f57f1119742bfbeea79fc5e59b6a08e282100fb62a4b320c56" Oct 29 23:30:19.477660 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.477534 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz"] Oct 29 23:30:19.478736 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.478717 6507 scope.go:122] "RemoveContainer" containerID="b8b669694b4318f57f1119742bfbeea79fc5e59b6a08e282100fb62a4b320c56" Oct 29 23:30:19.480001 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:19.479974 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b669694b4318f57f1119742bfbeea79fc5e59b6a08e282100fb62a4b320c56\": container with ID starting with b8b669694b4318f57f1119742bfbeea79fc5e59b6a08e282100fb62a4b320c56 not found: ID does not exist" containerID="b8b669694b4318f57f1119742bfbeea79fc5e59b6a08e282100fb62a4b320c56" Oct 29 23:30:19.480078 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.480007 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b669694b4318f57f1119742bfbeea79fc5e59b6a08e282100fb62a4b320c56"} err="failed to get container status \"b8b669694b4318f57f1119742bfbeea79fc5e59b6a08e282100fb62a4b320c56\": rpc error: code = NotFound desc = could not find container \"b8b669694b4318f57f1119742bfbeea79fc5e59b6a08e282100fb62a4b320c56\": container with ID starting with b8b669694b4318f57f1119742bfbeea79fc5e59b6a08e282100fb62a4b320c56 not found: ID does not exist" Oct 29 23:30:19.480078 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.480026 6507 scope.go:122] "RemoveContainer" containerID="c1c31919fa712acb0dbfc994634efe6f143fbe9caaba34b744c7a47db4519fb0" Oct 29 23:30:19.492026 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.491992 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2670a5-29888610-tgxzz"] Oct 29 23:30:19.495580 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.495556 6507 scope.go:122] "RemoveContainer" containerID="c1c31919fa712acb0dbfc994634efe6f143fbe9caaba34b744c7a47db4519fb0" Oct 29 23:30:19.497304 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:19.496920 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c31919fa712acb0dbfc994634efe6f143fbe9caaba34b744c7a47db4519fb0\": container with ID starting with c1c31919fa712acb0dbfc994634efe6f143fbe9caaba34b744c7a47db4519fb0 not found: ID does not exist" containerID="c1c31919fa712acb0dbfc994634efe6f143fbe9caaba34b744c7a47db4519fb0" Oct 29 23:30:19.497304 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.496959 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c31919fa712acb0dbfc994634efe6f143fbe9caaba34b744c7a47db4519fb0"} err="failed to get container status \"c1c31919fa712acb0dbfc994634efe6f143fbe9caaba34b744c7a47db4519fb0\": rpc error: code = NotFound desc = could not find container \"c1c31919fa712acb0dbfc994634efe6f143fbe9caaba34b744c7a47db4519fb0\": container with ID starting with c1c31919fa712acb0dbfc994634efe6f143fbe9caaba34b744c7a47db4519fb0 not found: ID does not exist" Oct 29 23:30:19.497304 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.496986 6507 scope.go:122] "RemoveContainer" containerID="291a70375f0f73d7788e008e0ab608bc39055e9765bda3c96d26c656db9b4736" Oct 29 23:30:19.497304 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.497298 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" podUID="10f9be75-9161-4cd4-9b75-8428406eed15" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:30:19.498749 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.498721 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" podUID="50c831dd-76a8-4e42-90b4-19847ae771e4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:30:19.507443 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.507415 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj"] Oct 29 23:30:19.510367 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.510347 6507 scope.go:122] "RemoveContainer" containerID="291a70375f0f73d7788e008e0ab608bc39055e9765bda3c96d26c656db9b4736" Oct 29 23:30:19.510745 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:19.510719 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291a70375f0f73d7788e008e0ab608bc39055e9765bda3c96d26c656db9b4736\": container with ID starting with 291a70375f0f73d7788e008e0ab608bc39055e9765bda3c96d26c656db9b4736 not found: ID does not exist" containerID="291a70375f0f73d7788e008e0ab608bc39055e9765bda3c96d26c656db9b4736" Oct 29 23:30:19.510827 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.510756 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291a70375f0f73d7788e008e0ab608bc39055e9765bda3c96d26c656db9b4736"} err="failed to get container status \"291a70375f0f73d7788e008e0ab608bc39055e9765bda3c96d26c656db9b4736\": rpc error: code = NotFound desc = could not find container \"291a70375f0f73d7788e008e0ab608bc39055e9765bda3c96d26c656db9b4736\": container with ID starting with 291a70375f0f73d7788e008e0ab608bc39055e9765bda3c96d26c656db9b4736 not found: ID does not exist" Oct 29 23:30:19.518024 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.517988 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg"] Oct 29 23:30:19.573599 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.573552 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p"] Oct 29 23:30:19.625811 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.609570 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn"] Oct 29 23:30:19.625811 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.615478 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-5xgwn"] Oct 29 23:30:19.646925 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.646863 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j"] Oct 29 23:30:19.658874 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.658569 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0b2ae9-29888610-mmq5j"] Oct 29 23:30:19.663784 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.663758 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" Oct 29 23:30:19.709025 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.708988 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc"] Oct 29 23:30:19.714372 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.714216 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-4crtc"] Oct 29 23:30:19.739919 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.739886 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf"] Oct 29 23:30:19.740265 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.740242 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd068bac-7513-4e03-ab75-5f6777222571-kube-api-access-pcdbh\" (UniqueName: \"kubernetes.io/projected/dd068bac-7513-4e03-ab75-5f6777222571-kube-api-access-pcdbh\") pod \"dd068bac-7513-4e03-ab75-5f6777222571\" (UID: \"dd068bac-7513-4e03-ab75-5f6777222571\") " Oct 29 23:30:19.740343 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.740306 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd068bac-7513-4e03-ab75-5f6777222571-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd068bac-7513-4e03-ab75-5f6777222571-maas-api-service-ca\") pod \"dd068bac-7513-4e03-ab75-5f6777222571\" (UID: \"dd068bac-7513-4e03-ab75-5f6777222571\") " Oct 29 23:30:19.740761 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.740712 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd068bac-7513-4e03-ab75-5f6777222571-maas-api-service-ca" pod "dd068bac-7513-4e03-ab75-5f6777222571" (UID: "dd068bac-7513-4e03-ab75-5f6777222571"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:19.741120 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.741101 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" Oct 29 23:30:19.741207 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.741131 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd068bac-7513-4e03-ab75-5f6777222571-maas-api-service-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.743347 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.743318 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd068bac-7513-4e03-ab75-5f6777222571-kube-api-access-pcdbh" pod "dd068bac-7513-4e03-ab75-5f6777222571" (UID: "dd068bac-7513-4e03-ab75-5f6777222571"). InnerVolumeSpecName "kube-api-access-pcdbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:19.745410 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.745386 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-jckbf"] Oct 29 23:30:19.779913 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.779757 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv"] Oct 29 23:30:19.788378 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.788346 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-313d9c-openshift-default-d98bdc757-djsxv"] Oct 29 23:30:19.860391 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.860359 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0f0d26-ae35-48ff-9ee6-d591f159ffcc-kube-api-access-5zbzz\" (UniqueName: \"kubernetes.io/projected/5f0f0d26-ae35-48ff-9ee6-d591f159ffcc-kube-api-access-5zbzz\") pod \"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc\" (UID: \"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc\") " Oct 29 23:30:19.860819 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.860795 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f0f0d26-ae35-48ff-9ee6-d591f159ffcc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f0f0d26-ae35-48ff-9ee6-d591f159ffcc-maas-api-service-ca\") pod \"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc\" (UID: \"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc\") " Oct 29 23:30:19.861170 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.861141 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0f0d26-ae35-48ff-9ee6-d591f159ffcc-maas-api-service-ca" pod "5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" (UID: "5f0f0d26-ae35-48ff-9ee6-d591f159ffcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:19.861756 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.861727 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f0f0d26-ae35-48ff-9ee6-d591f159ffcc-maas-api-service-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.861859 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.861761 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcdbh\" (UniqueName: \"kubernetes.io/projected/dd068bac-7513-4e03-ab75-5f6777222571-kube-api-access-pcdbh\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.872600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.870428 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0f0d26-ae35-48ff-9ee6-d591f159ffcc-kube-api-access-5zbzz" pod "5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" (UID: "5f0f0d26-ae35-48ff-9ee6-d591f159ffcc"). InnerVolumeSpecName "kube-api-access-5zbzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:19.962921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:19.962881 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zbzz\" (UniqueName: \"kubernetes.io/projected/5f0f0d26-ae35-48ff-9ee6-d591f159ffcc-kube-api-access-5zbzz\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:20.477350 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.477309 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" event={"ID":"dd068bac-7513-4e03-ab75-5f6777222571","Type":"ContainerDied","Data":"66e11db3ba8db271079445f3e41d37386021ce6c1ef68c655018bbc73b9a79d2"} Oct 29 23:30:20.477771 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.477666 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq" Oct 29 23:30:20.480540 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.477742 6507 scope.go:122] "RemoveContainer" containerID="d88cd9c438ac4d8d17624edb92ccbfd5109bfadf14c146045f6673cf2cb7ac39" Oct 29 23:30:20.483824 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.483612 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" event={"ID":"5f0f0d26-ae35-48ff-9ee6-d591f159ffcc","Type":"ContainerDied","Data":"acf3076c0e4e1f665e42cd930ee896a1499b643f6496a88195af29df87573c81"} Oct 29 23:30:20.484757 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.484743 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj" Oct 29 23:30:20.502655 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.502610 6507 scope.go:122] "RemoveContainer" containerID="e3dc67a93e7cd4b4d012e4ed4b7ee878be8e41e333d219ad2deb993e5ed18ec5" Oct 29 23:30:20.536839 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.536774 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq"] Oct 29 23:30:20.544213 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.544177 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-pk9cq"] Oct 29 23:30:20.577423 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.577382 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj"] Oct 29 23:30:20.581725 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.581692 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-032fa8-29888610-rxjpj"] Oct 29 23:30:20.754562 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.754529 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089367aa-c3e8-4f67-af00-697e5d7555bd" path="/var/lib/kubelet/pods/089367aa-c3e8-4f67-af00-697e5d7555bd/volumes" Oct 29 23:30:20.755034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.755017 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="185d4120-7013-49b4-af2d-0ec8aa4a7f14" path="/var/lib/kubelet/pods/185d4120-7013-49b4-af2d-0ec8aa4a7f14/volumes" Oct 29 23:30:20.755508 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.755492 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25fce51a-8d4e-4ac0-87e2-68457f9f10df" path="/var/lib/kubelet/pods/25fce51a-8d4e-4ac0-87e2-68457f9f10df/volumes" Oct 29 23:30:20.756197 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.756183 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a417439-524a-4ee1-ab88-3f083c09c179" path="/var/lib/kubelet/pods/3a417439-524a-4ee1-ab88-3f083c09c179/volumes" Oct 29 23:30:20.757764 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.757276 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" path="/var/lib/kubelet/pods/5f0f0d26-ae35-48ff-9ee6-d591f159ffcc/volumes" Oct 29 23:30:20.759354 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.758107 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914b9029-3376-4fad-addf-ce1b73391b5e" path="/var/lib/kubelet/pods/914b9029-3376-4fad-addf-ce1b73391b5e/volumes" Oct 29 23:30:20.759354 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.759227 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c272aaa1-73f3-4455-abe8-c61ca1047de3" path="/var/lib/kubelet/pods/c272aaa1-73f3-4455-abe8-c61ca1047de3/volumes" Oct 29 23:30:20.760137 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.759945 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd068bac-7513-4e03-ab75-5f6777222571" path="/var/lib/kubelet/pods/dd068bac-7513-4e03-ab75-5f6777222571/volumes" Oct 29 23:30:20.761781 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:20.760575 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" path="/var/lib/kubelet/pods/fa54faa3-3bb0-477e-9e21-a44b1d5a92b6/volumes" Oct 29 23:30:23.419468 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.419427 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp"] Oct 29 23:30:23.420242 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420217 6507 cpu_manager.go:414] "containerMap: removing container" podUID="5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" containerName="cleanup" Oct 29 23:30:23.420370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420244 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" containerName="cleanup" Oct 29 23:30:23.420370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420260 6507 cpu_manager.go:414] "containerMap: removing container" podUID="fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" containerName="cleanup" Oct 29 23:30:23.420370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420267 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" containerName="cleanup" Oct 29 23:30:23.420370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420281 6507 cpu_manager.go:414] "containerMap: removing container" podUID="185d4120-7013-49b4-af2d-0ec8aa4a7f14" containerName="cleanup" Oct 29 23:30:23.420370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420289 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185d4120-7013-49b4-af2d-0ec8aa4a7f14" containerName="cleanup" Oct 29 23:30:23.420370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420307 6507 cpu_manager.go:414] "containerMap: removing container" podUID="6173dedb-87b4-4028-9506-34ffadb93263" containerName="payload-processing" Oct 29 23:30:23.420370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420314 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6173dedb-87b4-4028-9506-34ffadb93263" containerName="payload-processing" Oct 29 23:30:23.420370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420326 6507 cpu_manager.go:414] "containerMap: removing container" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" Oct 29 23:30:23.420370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420332 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" Oct 29 23:30:23.420370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420343 6507 cpu_manager.go:414] "containerMap: removing container" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" Oct 29 23:30:23.420370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420351 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" Oct 29 23:30:23.420370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420364 6507 cpu_manager.go:414] "containerMap: removing container" podUID="3a417439-524a-4ee1-ab88-3f083c09c179" containerName="cleanup" Oct 29 23:30:23.420370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420371 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a417439-524a-4ee1-ab88-3f083c09c179" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420385 6507 cpu_manager.go:414] "containerMap: removing container" podUID="3a417439-524a-4ee1-ab88-3f083c09c179" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420392 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a417439-524a-4ee1-ab88-3f083c09c179" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420413 6507 cpu_manager.go:414] "containerMap: removing container" podUID="914b9029-3376-4fad-addf-ce1b73391b5e" containerName="istio-proxy" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420421 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914b9029-3376-4fad-addf-ce1b73391b5e" containerName="istio-proxy" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420434 6507 cpu_manager.go:414] "containerMap: removing container" podUID="25fce51a-8d4e-4ac0-87e2-68457f9f10df" containerName="istio-proxy" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420441 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25fce51a-8d4e-4ac0-87e2-68457f9f10df" containerName="istio-proxy" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420450 6507 cpu_manager.go:414] "containerMap: removing container" podUID="089367aa-c3e8-4f67-af00-697e5d7555bd" containerName="istio-proxy" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420472 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089367aa-c3e8-4f67-af00-697e5d7555bd" containerName="istio-proxy" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420489 6507 cpu_manager.go:414] "containerMap: removing container" podUID="3a417439-524a-4ee1-ab88-3f083c09c179" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420495 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a417439-524a-4ee1-ab88-3f083c09c179" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420510 6507 cpu_manager.go:414] "containerMap: removing container" podUID="9c730892-1170-44b8-9fcf-e74a43035722" containerName="payload-pre-processing" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420517 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c730892-1170-44b8-9fcf-e74a43035722" containerName="payload-pre-processing" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420530 6507 cpu_manager.go:414] "containerMap: removing container" podUID="c272aaa1-73f3-4455-abe8-c61ca1047de3" containerName="istio-proxy" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420537 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c272aaa1-73f3-4455-abe8-c61ca1047de3" containerName="istio-proxy" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420553 6507 cpu_manager.go:414] "containerMap: removing container" podUID="185d4120-7013-49b4-af2d-0ec8aa4a7f14" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420560 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185d4120-7013-49b4-af2d-0ec8aa4a7f14" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420570 6507 cpu_manager.go:414] "containerMap: removing container" podUID="5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420577 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420596 6507 cpu_manager.go:414] "containerMap: removing container" podUID="dd068bac-7513-4e03-ab75-5f6777222571" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420604 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd068bac-7513-4e03-ab75-5f6777222571" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420616 6507 cpu_manager.go:414] "containerMap: removing container" podUID="fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420623 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420641 6507 cpu_manager.go:414] "containerMap: removing container" podUID="dd068bac-7513-4e03-ab75-5f6777222571" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420649 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd068bac-7513-4e03-ab75-5f6777222571" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420664 6507 cpu_manager.go:414] "containerMap: removing container" podUID="dd068bac-7513-4e03-ab75-5f6777222571" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420671 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd068bac-7513-4e03-ab75-5f6777222571" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420828 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="25fce51a-8d4e-4ac0-87e2-68457f9f10df" containerName="istio-proxy" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420843 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="089367aa-c3e8-4f67-af00-697e5d7555bd" containerName="istio-proxy" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420853 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a417439-524a-4ee1-ab88-3f083c09c179" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420862 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c730892-1170-44b8-9fcf-e74a43035722" containerName="payload-pre-processing" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420874 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="c272aaa1-73f3-4455-abe8-c61ca1047de3" containerName="istio-proxy" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420886 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="185d4120-7013-49b4-af2d-0ec8aa4a7f14" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420894 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420904 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420913 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd068bac-7513-4e03-ab75-5f6777222571" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420924 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420933 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd068bac-7513-4e03-ab75-5f6777222571" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420941 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="185d4120-7013-49b4-af2d-0ec8aa4a7f14" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420950 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd068bac-7513-4e03-ab75-5f6777222571" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420959 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420951 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" podUID="56b5ecf5-e109-433c-963d-bf97a5bd6f8a" containerName="payload-processing" containerID="cri-o://cf43714541cc6be5c9272430fee0b8b1f373d7038cd504c588f67aad97f3d52a" gracePeriod=25 Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.420969 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="185d4120-7013-49b4-af2d-0ec8aa4a7f14" containerName="cleanup" Oct 29 23:30:23.421034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.421060 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="6173dedb-87b4-4028-9506-34ffadb93263" containerName="payload-processing" Oct 29 23:30:23.423086 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.421070 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fb6a01-f848-4ae7-86dc-26e45162b41c" containerName="main" Oct 29 23:30:23.423086 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.421081 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="045c37c8-57a0-4ed1-87a6-89f73eb079d2" containerName="main" Oct 29 23:30:23.423086 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.421089 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a417439-524a-4ee1-ab88-3f083c09c179" containerName="cleanup" Oct 29 23:30:23.423086 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.421099 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" containerName="cleanup" Oct 29 23:30:23.423086 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.421109 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="914b9029-3376-4fad-addf-ce1b73391b5e" containerName="istio-proxy" Oct 29 23:30:23.427024 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.427001 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" Oct 29 23:30:23.450234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.450158 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:30:23.450234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.450172 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:30:23.450522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.450490 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:30:23.450697 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.450673 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:30:23.453548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.453521 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp"] Oct 29 23:30:23.473752 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.473707 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" podUID="8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26" containerName="payload-pre-processing" containerID="cri-o://90da7b065cc3fdb3456ce9da4ca0713123a51c97766fc4b481522497c07c9d05" gracePeriod=25 Oct 29 23:30:23.552895 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.552864 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xwv5\" (UniqueName: \"kubernetes.io/projected/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-kube-api-access-5xwv5\") pod \"maas-api-revoke-sub-56c27fd0-gz4sp\" (UID: \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" Oct 29 23:30:23.553058 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.552932 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56c27fd0-gz4sp\" (UID: \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" Oct 29 23:30:23.553058 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.553030 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56c27fd0-gz4sp\" (UID: \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" Oct 29 23:30:23.654904 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.654864 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56c27fd0-gz4sp\" (UID: \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" Oct 29 23:30:23.655098 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.654912 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56c27fd0-gz4sp\" (UID: \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" Oct 29 23:30:23.655181 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.655161 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xwv5\" (UniqueName: \"kubernetes.io/projected/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-kube-api-access-5xwv5\") pod \"maas-api-revoke-sub-56c27fd0-gz4sp\" (UID: \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" Oct 29 23:30:23.655629 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.655608 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56c27fd0-gz4sp\" (UID: \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" Oct 29 23:30:23.657359 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.657333 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56c27fd0-gz4sp\" (UID: \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" Oct 29 23:30:23.671962 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.671906 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xwv5\" (UniqueName: \"kubernetes.io/projected/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-kube-api-access-5xwv5\") pod \"maas-api-revoke-sub-56c27fd0-gz4sp\" (UID: \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" Oct 29 23:30:23.741683 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:23.741647 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" Oct 29 23:30:24.046299 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:24.046268 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp"] Oct 29 23:30:24.048189 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:30:24.048161 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aecafa4_7d6e_4fa5_adc8_05e37a772b0e.slice/crio-948efc9bc1b655817535d6caaa2583bfb94848876efe8c54111c024744666509 WatchSource:0}: Error finding container 948efc9bc1b655817535d6caaa2583bfb94848876efe8c54111c024744666509: Status 404 returned error can't find the container with id 948efc9bc1b655817535d6caaa2583bfb94848876efe8c54111c024744666509 Oct 29 23:30:24.518559 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:24.518517 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" podUID="71038474-e817-4ff1-be48-145e6f10cb3c" containerName="payload-processing" containerID="cri-o://b23794173ebc514c6f2e63ad497e1aa15c01986f8039a44e2e63961c5398b5cd" gracePeriod=25 Oct 29 23:30:24.537686 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:24.537643 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" event={"ID":"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e","Type":"ContainerStarted","Data":"e20754ef5d4677ddac128d905bb199243b78f2bdb324b1a08ea1fc9352c1f0cf"} Oct 29 23:30:24.537862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:24.537711 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" event={"ID":"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e","Type":"ContainerStarted","Data":"948efc9bc1b655817535d6caaa2583bfb94848876efe8c54111c024744666509"} Oct 29 23:30:24.574264 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:24.574157 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" podStartSLOduration=1.574140594 podStartE2EDuration="1.574140594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:24.570292982 +0000 UTC m=+993.933297704" watchObservedRunningTime="2026-10-29 23:30:24.574140594 +0000 UTC m=+993.937145317" Oct 29 23:30:24.574426 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:24.574352 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" podUID="cb462872-9968-4e91-96c9-675ffd093457" containerName="payload-pre-processing" containerID="cri-o://750047c8bae361f192f8e2f5e12aec9a9bb1fff05002b40f98186bf16b0fb282" gracePeriod=25 Oct 29 23:30:25.547633 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:25.547601 6507 generic.go:356] "Generic (PLEG): container finished" podID="4aecafa4-7d6e-4fa5-adc8-05e37a772b0e" containerID="e20754ef5d4677ddac128d905bb199243b78f2bdb324b1a08ea1fc9352c1f0cf" exitCode=0 Oct 29 23:30:25.548036 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:25.547688 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" event={"ID":"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e","Type":"ContainerDied","Data":"e20754ef5d4677ddac128d905bb199243b78f2bdb324b1a08ea1fc9352c1f0cf"} Oct 29 23:30:26.568650 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.568616 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn"] Oct 29 23:30:26.569496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.569451 6507 cpu_manager.go:414] "containerMap: removing container" podUID="5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" containerName="cleanup" Oct 29 23:30:26.569496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.569491 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0f0d26-ae35-48ff-9ee6-d591f159ffcc" containerName="cleanup" Oct 29 23:30:26.569679 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.569508 6507 cpu_manager.go:414] "containerMap: removing container" podUID="fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" containerName="cleanup" Oct 29 23:30:26.569679 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.569517 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" containerName="cleanup" Oct 29 23:30:26.569679 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.569528 6507 cpu_manager.go:414] "containerMap: removing container" podUID="185d4120-7013-49b4-af2d-0ec8aa4a7f14" containerName="cleanup" Oct 29 23:30:26.569679 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.569536 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185d4120-7013-49b4-af2d-0ec8aa4a7f14" containerName="cleanup" Oct 29 23:30:26.569679 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.569665 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a417439-524a-4ee1-ab88-3f083c09c179" containerName="cleanup" Oct 29 23:30:26.569850 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.569683 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa54faa3-3bb0-477e-9e21-a44b1d5a92b6" containerName="cleanup" Oct 29 23:30:26.573351 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.573330 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" Oct 29 23:30:26.605144 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.604985 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn"] Oct 29 23:30:26.705885 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.705847 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63b88d11-kmgxn\" (UID: \"60ba8a59-d4e6-4d18-ba32-3c6641e543c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" Oct 29 23:30:26.705885 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.705887 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72mh5\" (UniqueName: \"kubernetes.io/projected/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-kube-api-access-72mh5\") pod \"maas-api-revoke-sub-63b88d11-kmgxn\" (UID: \"60ba8a59-d4e6-4d18-ba32-3c6641e543c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" Oct 29 23:30:26.706178 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.706118 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63b88d11-kmgxn\" (UID: \"60ba8a59-d4e6-4d18-ba32-3c6641e543c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" Oct 29 23:30:26.763550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.763516 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" Oct 29 23:30:26.826534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.823705 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63b88d11-kmgxn\" (UID: \"60ba8a59-d4e6-4d18-ba32-3c6641e543c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" Oct 29 23:30:26.826534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.823966 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63b88d11-kmgxn\" (UID: \"60ba8a59-d4e6-4d18-ba32-3c6641e543c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" Oct 29 23:30:26.826534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.824001 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72mh5\" (UniqueName: \"kubernetes.io/projected/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-kube-api-access-72mh5\") pod \"maas-api-revoke-sub-63b88d11-kmgxn\" (UID: \"60ba8a59-d4e6-4d18-ba32-3c6641e543c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" Oct 29 23:30:26.826534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.826012 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63b88d11-kmgxn\" (UID: \"60ba8a59-d4e6-4d18-ba32-3c6641e543c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" Oct 29 23:30:26.926814 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.926779 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-maas-api-xks-ca\") pod \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\" (UID: \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\") " Oct 29 23:30:26.927794 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.927768 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-maas-api-openshift-ca" pod "4aecafa4-7d6e-4fa5-adc8-05e37a772b0e" (UID: "4aecafa4-7d6e-4fa5-adc8-05e37a772b0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:26.927956 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.927941 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-maas-api-openshift-ca\") pod \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\" (UID: \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\") " Oct 29 23:30:26.928340 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.928324 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-kube-api-access-5xwv5\" (UniqueName: \"kubernetes.io/projected/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-kube-api-access-5xwv5\") pod \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\" (UID: \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\") " Oct 29 23:30:26.929386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.929369 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:26.957480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:26.936175 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72mh5\" (UniqueName: \"kubernetes.io/projected/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-kube-api-access-72mh5\") pod \"maas-api-revoke-sub-63b88d11-kmgxn\" (UID: \"60ba8a59-d4e6-4d18-ba32-3c6641e543c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" Oct 29 23:30:27.079529 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.050697 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63b88d11-kmgxn\" (UID: \"60ba8a59-d4e6-4d18-ba32-3c6641e543c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" Oct 29 23:30:27.080665 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.080623 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-maas-api-xks-ca" pod "4aecafa4-7d6e-4fa5-adc8-05e37a772b0e" (UID: "4aecafa4-7d6e-4fa5-adc8-05e37a772b0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:30:27.243508 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.234775 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-kube-api-access-5xwv5" pod "4aecafa4-7d6e-4fa5-adc8-05e37a772b0e" (UID: "4aecafa4-7d6e-4fa5-adc8-05e37a772b0e"). InnerVolumeSpecName "kube-api-access-5xwv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:27.247034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.246952 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" Oct 29 23:30:27.259550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.259510 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd"] Oct 29 23:30:27.260809 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.260787 6507 cpu_manager.go:414] "containerMap: removing container" podUID="4aecafa4-7d6e-4fa5-adc8-05e37a772b0e" containerName="revoke-keys" Oct 29 23:30:27.260934 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.260921 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aecafa4-7d6e-4fa5-adc8-05e37a772b0e" containerName="revoke-keys" Oct 29 23:30:27.261196 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.261184 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aecafa4-7d6e-4fa5-adc8-05e37a772b0e" containerName="revoke-keys" Oct 29 23:30:27.271442 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.265774 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd"] Oct 29 23:30:27.284491 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.272418 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" Oct 29 23:30:27.285144 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.283604 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-kube-api-access-5xwv5\" (UniqueName: \"kubernetes.io/projected/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-kube-api-access-5xwv5\") pod \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\" (UID: \"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e\") " Oct 29 23:30:27.308630 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.285926 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:27.308630 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:30:27.283724 6507 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e/volumes/kubernetes.io~projected/kube-api-access-5xwv5 Oct 29 23:30:27.308630 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.288931 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-kube-api-access-5xwv5" pod "4aecafa4-7d6e-4fa5-adc8-05e37a772b0e" (UID: "4aecafa4-7d6e-4fa5-adc8-05e37a772b0e"). InnerVolumeSpecName "kube-api-access-5xwv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:27.389635 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.386167 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" containerID="cri-o://ce3a8f265d66811d4ed0daf1a57a6b541eeca3d513c2129053f8c267860b34fa" gracePeriod=45 Oct 29 23:30:27.389635 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.387551 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73c49368-4334-4699-99dc-a1c19dbc9fce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aaf95ce-zkkxd\" (UID: \"73c49368-4334-4699-99dc-a1c19dbc9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" Oct 29 23:30:27.389635 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.387671 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8j4s\" (UniqueName: \"kubernetes.io/projected/73c49368-4334-4699-99dc-a1c19dbc9fce-kube-api-access-k8j4s\") pod \"maas-api-revoke-sub-6aaf95ce-zkkxd\" (UID: \"73c49368-4334-4699-99dc-a1c19dbc9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" Oct 29 23:30:27.389635 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.387706 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73c49368-4334-4699-99dc-a1c19dbc9fce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aaf95ce-zkkxd\" (UID: \"73c49368-4334-4699-99dc-a1c19dbc9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" Oct 29 23:30:27.389635 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.387836 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xwv5\" (UniqueName: \"kubernetes.io/projected/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e-kube-api-access-5xwv5\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:27.439371 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.439277 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6"] Oct 29 23:30:27.445187 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.445155 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-032fa8-h5dp6"] Oct 29 23:30:27.489400 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.489361 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73c49368-4334-4699-99dc-a1c19dbc9fce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aaf95ce-zkkxd\" (UID: \"73c49368-4334-4699-99dc-a1c19dbc9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" Oct 29 23:30:27.489606 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.489491 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8j4s\" (UniqueName: \"kubernetes.io/projected/73c49368-4334-4699-99dc-a1c19dbc9fce-kube-api-access-k8j4s\") pod \"maas-api-revoke-sub-6aaf95ce-zkkxd\" (UID: \"73c49368-4334-4699-99dc-a1c19dbc9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" Oct 29 23:30:27.489606 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.489524 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73c49368-4334-4699-99dc-a1c19dbc9fce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aaf95ce-zkkxd\" (UID: \"73c49368-4334-4699-99dc-a1c19dbc9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" Oct 29 23:30:27.490239 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.490211 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73c49368-4334-4699-99dc-a1c19dbc9fce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aaf95ce-zkkxd\" (UID: \"73c49368-4334-4699-99dc-a1c19dbc9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" Oct 29 23:30:27.492151 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.492126 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73c49368-4334-4699-99dc-a1c19dbc9fce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aaf95ce-zkkxd\" (UID: \"73c49368-4334-4699-99dc-a1c19dbc9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" Oct 29 23:30:27.507366 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.507337 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8j4s\" (UniqueName: \"kubernetes.io/projected/73c49368-4334-4699-99dc-a1c19dbc9fce-kube-api-access-k8j4s\") pod \"maas-api-revoke-sub-6aaf95ce-zkkxd\" (UID: \"73c49368-4334-4699-99dc-a1c19dbc9fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" Oct 29 23:30:27.558970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.558591 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": dial tcp 10.133.0.71:8000: connect: connection refused" Oct 29 23:30:27.649987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.649678 6507 generic.go:356] "Generic (PLEG): container finished" podID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerID="ce3a8f265d66811d4ed0daf1a57a6b541eeca3d513c2129053f8c267860b34fa" exitCode=0 Oct 29 23:30:27.649987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.649816 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" event={"ID":"87843995-cc71-42f6-ac3d-a5f3b13907c0","Type":"ContainerDied","Data":"ce3a8f265d66811d4ed0daf1a57a6b541eeca3d513c2129053f8c267860b34fa"} Oct 29 23:30:27.652991 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.652959 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" event={"ID":"4aecafa4-7d6e-4fa5-adc8-05e37a772b0e","Type":"ContainerDied","Data":"948efc9bc1b655817535d6caaa2583bfb94848876efe8c54111c024744666509"} Oct 29 23:30:27.653338 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.653323 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="948efc9bc1b655817535d6caaa2583bfb94848876efe8c54111c024744666509" Oct 29 23:30:27.653475 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.653208 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp" Oct 29 23:30:27.701240 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.701141 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" Oct 29 23:30:27.706350 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.706309 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn"] Oct 29 23:30:27.708110 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:30:27.708079 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ba8a59_d4e6_4d18_ba32_3c6641e543c4.slice/crio-b2b19d47b718b2c3044105667ee6562f931e6465f16c6c5db1ccff6a7010c610 WatchSource:0}: Error finding container b2b19d47b718b2c3044105667ee6562f931e6465f16c6c5db1ccff6a7010c610: Status 404 returned error can't find the container with id b2b19d47b718b2c3044105667ee6562f931e6465f16c6c5db1ccff6a7010c610 Oct 29 23:30:27.847476 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:27.847436 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:30:28.008216 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.008175 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-model-cache\") pod \"87843995-cc71-42f6-ac3d-a5f3b13907c0\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " Oct 29 23:30:28.008379 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.008233 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-home\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-home\") pod \"87843995-cc71-42f6-ac3d-a5f3b13907c0\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " Oct 29 23:30:28.008379 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.008352 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-dshm\") pod \"87843995-cc71-42f6-ac3d-a5f3b13907c0\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " Oct 29 23:30:28.008500 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.008432 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-model-cache" pod "87843995-cc71-42f6-ac3d-a5f3b13907c0" (UID: "87843995-cc71-42f6-ac3d-a5f3b13907c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:28.008605 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.008583 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-home" pod "87843995-cc71-42f6-ac3d-a5f3b13907c0" (UID: "87843995-cc71-42f6-ac3d-a5f3b13907c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:28.008807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.008784 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87843995-cc71-42f6-ac3d-a5f3b13907c0-kube-api-access-v424c\" (UniqueName: \"kubernetes.io/projected/87843995-cc71-42f6-ac3d-a5f3b13907c0-kube-api-access-v424c\") pod \"87843995-cc71-42f6-ac3d-a5f3b13907c0\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " Oct 29 23:30:28.008941 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.008816 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87843995-cc71-42f6-ac3d-a5f3b13907c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/87843995-cc71-42f6-ac3d-a5f3b13907c0-tls-certs\") pod \"87843995-cc71-42f6-ac3d-a5f3b13907c0\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " Oct 29 23:30:28.009004 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.008949 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-tmp-dir\") pod \"87843995-cc71-42f6-ac3d-a5f3b13907c0\" (UID: \"87843995-cc71-42f6-ac3d-a5f3b13907c0\") " Oct 29 23:30:28.009394 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.009352 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-tmp-dir" pod "87843995-cc71-42f6-ac3d-a5f3b13907c0" (UID: "87843995-cc71-42f6-ac3d-a5f3b13907c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:28.009977 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.009652 6507 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-model-cache\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:28.009977 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.009671 6507 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-home\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:28.009977 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.009686 6507 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-tmp-dir\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:28.010663 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.010631 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-dshm" pod "87843995-cc71-42f6-ac3d-a5f3b13907c0" (UID: "87843995-cc71-42f6-ac3d-a5f3b13907c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:28.010967 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.010949 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87843995-cc71-42f6-ac3d-a5f3b13907c0-kube-api-access-v424c" pod "87843995-cc71-42f6-ac3d-a5f3b13907c0" (UID: "87843995-cc71-42f6-ac3d-a5f3b13907c0"). InnerVolumeSpecName "kube-api-access-v424c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:28.011210 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.011178 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87843995-cc71-42f6-ac3d-a5f3b13907c0-tls-certs" pod "87843995-cc71-42f6-ac3d-a5f3b13907c0" (UID: "87843995-cc71-42f6-ac3d-a5f3b13907c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:30:28.049842 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.049817 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd"] Oct 29 23:30:28.051057 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:30:28.051033 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c49368_4334_4699_99dc_a1c19dbc9fce.slice/crio-3d8dd7279a7cc68910f40d335846cf971b0e31cf9a00d367fdb492a42c821da1 WatchSource:0}: Error finding container 3d8dd7279a7cc68910f40d335846cf971b0e31cf9a00d367fdb492a42c821da1: Status 404 returned error can't find the container with id 3d8dd7279a7cc68910f40d335846cf971b0e31cf9a00d367fdb492a42c821da1 Oct 29 23:30:28.111366 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.111330 6507 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87843995-cc71-42f6-ac3d-a5f3b13907c0-dshm\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:28.111366 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.111362 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v424c\" (UniqueName: \"kubernetes.io/projected/87843995-cc71-42f6-ac3d-a5f3b13907c0-kube-api-access-v424c\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:28.111366 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.111373 6507 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87843995-cc71-42f6-ac3d-a5f3b13907c0-tls-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:28.607504 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.607438 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" containerID="cri-o://4817da9f0cc3a031e8d9134c698a5ea46c41e9db506f76343d3f318a85e8d444" gracePeriod=45 Oct 29 23:30:28.616427 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.616390 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" containerID="cri-o://1510a821cfcab096663700e38dae9f8deb17b5da76daa6ff2cb3bc4067d7936e" gracePeriod=45 Oct 29 23:30:28.662211 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.662181 6507 generic.go:356] "Generic (PLEG): container finished" podID="10f9be75-9161-4cd4-9b75-8428406eed15" containerID="3cc034b607740335b7b1c3dbdbdb80110f3640cb10d1bc98d4e6bd3a6a41bda3" exitCode=0 Oct 29 23:30:28.662596 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.662251 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" event={"ID":"10f9be75-9161-4cd4-9b75-8428406eed15","Type":"ContainerDied","Data":"3cc034b607740335b7b1c3dbdbdb80110f3640cb10d1bc98d4e6bd3a6a41bda3"} Oct 29 23:30:28.664059 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.664035 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" event={"ID":"73c49368-4334-4699-99dc-a1c19dbc9fce","Type":"ContainerStarted","Data":"b9b9a0ff88ef18df461ff7699668334ca12a188bfb1619ca12e5e736f3c684d0"} Oct 29 23:30:28.664184 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.664088 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" event={"ID":"73c49368-4334-4699-99dc-a1c19dbc9fce","Type":"ContainerStarted","Data":"3d8dd7279a7cc68910f40d335846cf971b0e31cf9a00d367fdb492a42c821da1"} Oct 29 23:30:28.665501 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.665452 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" event={"ID":"87843995-cc71-42f6-ac3d-a5f3b13907c0","Type":"ContainerDied","Data":"1c4be036d4d88306928d8337e10667cf30064b631bc2f41cee2d3adb001cec2e"} Oct 29 23:30:28.665580 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.665451 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k" Oct 29 23:30:28.665652 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.665509 6507 scope.go:122] "RemoveContainer" containerID="ce3a8f265d66811d4ed0daf1a57a6b541eeca3d513c2129053f8c267860b34fa" Oct 29 23:30:28.667135 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.667107 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" event={"ID":"60ba8a59-d4e6-4d18-ba32-3c6641e543c4","Type":"ContainerStarted","Data":"77168e06a14518aae60f8718992b888a0791e1b37eda1cdb613dbd77fc9588ce"} Oct 29 23:30:28.667203 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.667167 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" event={"ID":"60ba8a59-d4e6-4d18-ba32-3c6641e543c4","Type":"ContainerStarted","Data":"b2b19d47b718b2c3044105667ee6562f931e6465f16c6c5db1ccff6a7010c610"} Oct 29 23:30:28.706385 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.706335 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" podStartSLOduration=2.706319368 podStartE2EDuration="2.706319368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:28.701927203 +0000 UTC m=+998.064931925" watchObservedRunningTime="2026-10-29 23:30:28.706319368 +0000 UTC m=+998.069324091" Oct 29 23:30:28.777476 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.771070 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d767e464-9ffc-412c-a15f-f815f7a4caa0" path="/var/lib/kubelet/pods/d767e464-9ffc-412c-a15f-f815f7a4caa0/volumes" Oct 29 23:30:28.844576 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.843916 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" podStartSLOduration=2.8438970169999997 podStartE2EDuration="2.843897017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:28.756036627 +0000 UTC m=+998.119041349" watchObservedRunningTime="2026-10-29 23:30:28.843897017 +0000 UTC m=+998.206901741" Oct 29 23:30:28.851875 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.851819 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k"] Oct 29 23:30:28.858024 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.857986 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2670a5/distinct-w0-2670a5-kserve-579bf9ff4d-sv95k"] Oct 29 23:30:28.905659 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.905624 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" Oct 29 23:30:28.959771 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:28.959737 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" Oct 29 23:30:29.001874 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.001820 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:30:29.022916 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.022791 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:30:29.026094 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.026061 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c831dd-76a8-4e42-90b4-19847ae771e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c831dd-76a8-4e42-90b4-19847ae771e4-plugins-config-volume\") pod \"50c831dd-76a8-4e42-90b4-19847ae771e4\" (UID: \"50c831dd-76a8-4e42-90b4-19847ae771e4\") " Oct 29 23:30:29.026398 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.026379 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c831dd-76a8-4e42-90b4-19847ae771e4-kube-api-access-n596b\" (UniqueName: \"kubernetes.io/projected/50c831dd-76a8-4e42-90b4-19847ae771e4-kube-api-access-n596b\") pod \"50c831dd-76a8-4e42-90b4-19847ae771e4\" (UID: \"50c831dd-76a8-4e42-90b4-19847ae771e4\") " Oct 29 23:30:29.026491 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.026470 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c831dd-76a8-4e42-90b4-19847ae771e4-plugins-config-volume" pod "50c831dd-76a8-4e42-90b4-19847ae771e4" (UID: "50c831dd-76a8-4e42-90b4-19847ae771e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:29.026914 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.026895 6507 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50c831dd-76a8-4e42-90b4-19847ae771e4-plugins-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.028500 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.028430 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c831dd-76a8-4e42-90b4-19847ae771e4-kube-api-access-n596b" pod "50c831dd-76a8-4e42-90b4-19847ae771e4" (UID: "50c831dd-76a8-4e42-90b4-19847ae771e4"). InnerVolumeSpecName "kube-api-access-n596b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:29.128094 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128011 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-dshm\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-dshm\") pod \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " Oct 29 23:30:29.128094 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128050 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-home\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-home\") pod \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " Oct 29 23:30:29.128286 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128104 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-tmp-dir\") pod \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " Oct 29 23:30:29.128286 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128124 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-model-cache\") pod \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " Oct 29 23:30:29.128286 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128187 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f9be75-9161-4cd4-9b75-8428406eed15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f9be75-9161-4cd4-9b75-8428406eed15-plugins-config-volume\") pod \"10f9be75-9161-4cd4-9b75-8428406eed15\" (UID: \"10f9be75-9161-4cd4-9b75-8428406eed15\") " Oct 29 23:30:29.128286 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128238 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-tmp-dir\") pod \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " Oct 29 23:30:29.128286 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128278 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f9be75-9161-4cd4-9b75-8428406eed15-kube-api-access-n8sk5\" (UniqueName: \"kubernetes.io/projected/10f9be75-9161-4cd4-9b75-8428406eed15-kube-api-access-n8sk5\") pod \"10f9be75-9161-4cd4-9b75-8428406eed15\" (UID: \"10f9be75-9161-4cd4-9b75-8428406eed15\") " Oct 29 23:30:29.128554 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128309 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-home\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-home\") pod \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " Oct 29 23:30:29.128554 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128371 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-kube-api-access-nmdwc\" (UniqueName: \"kubernetes.io/projected/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-kube-api-access-nmdwc\") pod \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " Oct 29 23:30:29.128554 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128387 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-home" pod "bb0c6b55-eae3-4617-a503-ab72e111c5fa" (UID: "bb0c6b55-eae3-4617-a503-ab72e111c5fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:29.128554 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128388 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-tmp-dir" pod "24364d6f-9a22-4b9c-b2ff-47bca2cbba80" (UID: "24364d6f-9a22-4b9c-b2ff-47bca2cbba80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:29.128554 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128401 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-dshm\") pod \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " Oct 29 23:30:29.128554 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128439 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-model-cache" pod "24364d6f-9a22-4b9c-b2ff-47bca2cbba80" (UID: "24364d6f-9a22-4b9c-b2ff-47bca2cbba80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:29.128554 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128511 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-tls-certs\" (UniqueName: \"kubernetes.io/secret/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-tls-certs\") pod \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\" (UID: \"24364d6f-9a22-4b9c-b2ff-47bca2cbba80\") " Oct 29 23:30:29.128906 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128593 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-tmp-dir" pod "bb0c6b55-eae3-4617-a503-ab72e111c5fa" (UID: "bb0c6b55-eae3-4617-a503-ab72e111c5fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:29.128906 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128620 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-home" pod "24364d6f-9a22-4b9c-b2ff-47bca2cbba80" (UID: "24364d6f-9a22-4b9c-b2ff-47bca2cbba80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:29.128906 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128661 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb0c6b55-eae3-4617-a503-ab72e111c5fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb0c6b55-eae3-4617-a503-ab72e111c5fa-tls-certs\") pod \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " Oct 29 23:30:29.128906 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128664 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f9be75-9161-4cd4-9b75-8428406eed15-plugins-config-volume" pod "10f9be75-9161-4cd4-9b75-8428406eed15" (UID: "10f9be75-9161-4cd4-9b75-8428406eed15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:29.128906 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128789 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb0c6b55-eae3-4617-a503-ab72e111c5fa-kube-api-access-tc9f6\" (UniqueName: \"kubernetes.io/projected/bb0c6b55-eae3-4617-a503-ab72e111c5fa-kube-api-access-tc9f6\") pod \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " Oct 29 23:30:29.128906 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.128887 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-model-cache\") pod \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\" (UID: \"bb0c6b55-eae3-4617-a503-ab72e111c5fa\") " Oct 29 23:30:29.129948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.129629 6507 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-tmp-dir\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.129948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.129658 6507 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-home\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.129948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.129672 6507 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-home\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.129948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.129686 6507 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-tmp-dir\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.129948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.129699 6507 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-model-cache\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.129948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.129713 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n596b\" (UniqueName: \"kubernetes.io/projected/50c831dd-76a8-4e42-90b4-19847ae771e4-kube-api-access-n596b\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.129948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.129727 6507 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f9be75-9161-4cd4-9b75-8428406eed15-plugins-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.130593 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.130518 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-model-cache" pod "bb0c6b55-eae3-4617-a503-ab72e111c5fa" (UID: "bb0c6b55-eae3-4617-a503-ab72e111c5fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:29.131069 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.131039 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-dshm" pod "24364d6f-9a22-4b9c-b2ff-47bca2cbba80" (UID: "24364d6f-9a22-4b9c-b2ff-47bca2cbba80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:29.131231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.131084 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f9be75-9161-4cd4-9b75-8428406eed15-kube-api-access-n8sk5" pod "10f9be75-9161-4cd4-9b75-8428406eed15" (UID: "10f9be75-9161-4cd4-9b75-8428406eed15"). InnerVolumeSpecName "kube-api-access-n8sk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:29.131231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.131153 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb0c6b55-eae3-4617-a503-ab72e111c5fa-tls-certs" pod "bb0c6b55-eae3-4617-a503-ab72e111c5fa" (UID: "bb0c6b55-eae3-4617-a503-ab72e111c5fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:30:29.131231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.131173 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-tls-certs" pod "24364d6f-9a22-4b9c-b2ff-47bca2cbba80" (UID: "24364d6f-9a22-4b9c-b2ff-47bca2cbba80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:30:29.131231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.131211 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-dshm" pod "bb0c6b55-eae3-4617-a503-ab72e111c5fa" (UID: "bb0c6b55-eae3-4617-a503-ab72e111c5fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:29.131408 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.131388 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0c6b55-eae3-4617-a503-ab72e111c5fa-kube-api-access-tc9f6" pod "bb0c6b55-eae3-4617-a503-ab72e111c5fa" (UID: "bb0c6b55-eae3-4617-a503-ab72e111c5fa"). InnerVolumeSpecName "kube-api-access-tc9f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:29.131523 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.131442 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-kube-api-access-nmdwc" pod "24364d6f-9a22-4b9c-b2ff-47bca2cbba80" (UID: "24364d6f-9a22-4b9c-b2ff-47bca2cbba80"). InnerVolumeSpecName "kube-api-access-nmdwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:29.231590 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.231552 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmdwc\" (UniqueName: \"kubernetes.io/projected/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-kube-api-access-nmdwc\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.231590 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.231583 6507 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-dshm\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.231590 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.231594 6507 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-tls-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.231824 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.231602 6507 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb0c6b55-eae3-4617-a503-ab72e111c5fa-tls-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.231824 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.231611 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc9f6\" (UniqueName: \"kubernetes.io/projected/bb0c6b55-eae3-4617-a503-ab72e111c5fa-kube-api-access-tc9f6\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.231824 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.231618 6507 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb0c6b55-eae3-4617-a503-ab72e111c5fa-model-cache\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.231824 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.231628 6507 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24364d6f-9a22-4b9c-b2ff-47bca2cbba80-dshm\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.231824 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.231636 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8sk5\" (UniqueName: \"kubernetes.io/projected/10f9be75-9161-4cd4-9b75-8428406eed15-kube-api-access-n8sk5\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:29.488804 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.488758 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" podUID="cb462872-9968-4e91-96c9-675ffd093457" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:30:29.488971 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.488850 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" podUID="8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:30:29.489400 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.489379 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" podUID="56b5ecf5-e109-433c-963d-bf97a5bd6f8a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:30:29.677884 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.677852 6507 generic.go:356] "Generic (PLEG): container finished" podID="73c49368-4334-4699-99dc-a1c19dbc9fce" containerID="b9b9a0ff88ef18df461ff7699668334ca12a188bfb1619ca12e5e736f3c684d0" exitCode=0 Oct 29 23:30:29.678338 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.677944 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" event={"ID":"73c49368-4334-4699-99dc-a1c19dbc9fce","Type":"ContainerDied","Data":"b9b9a0ff88ef18df461ff7699668334ca12a188bfb1619ca12e5e736f3c684d0"} Oct 29 23:30:29.679313 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.679290 6507 generic.go:356] "Generic (PLEG): container finished" podID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerID="4817da9f0cc3a031e8d9134c698a5ea46c41e9db506f76343d3f318a85e8d444" exitCode=0 Oct 29 23:30:29.679429 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.679328 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" Oct 29 23:30:29.679429 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.679373 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" event={"ID":"bb0c6b55-eae3-4617-a503-ab72e111c5fa","Type":"ContainerDied","Data":"4817da9f0cc3a031e8d9134c698a5ea46c41e9db506f76343d3f318a85e8d444"} Oct 29 23:30:29.679429 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.679410 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k" event={"ID":"bb0c6b55-eae3-4617-a503-ab72e111c5fa","Type":"ContainerDied","Data":"f858fc8dbb3ce5ee5f30628a7e2c2ac0c242b5977abc06c8f820b6967e9d9a64"} Oct 29 23:30:29.679605 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.679431 6507 scope.go:122] "RemoveContainer" containerID="4817da9f0cc3a031e8d9134c698a5ea46c41e9db506f76343d3f318a85e8d444" Oct 29 23:30:29.680863 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.680845 6507 generic.go:356] "Generic (PLEG): container finished" podID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerID="1510a821cfcab096663700e38dae9f8deb17b5da76daa6ff2cb3bc4067d7936e" exitCode=0 Oct 29 23:30:29.680950 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.680905 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" Oct 29 23:30:29.680950 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.680918 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" event={"ID":"24364d6f-9a22-4b9c-b2ff-47bca2cbba80","Type":"ContainerDied","Data":"1510a821cfcab096663700e38dae9f8deb17b5da76daa6ff2cb3bc4067d7936e"} Oct 29 23:30:29.680950 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.680946 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr" event={"ID":"24364d6f-9a22-4b9c-b2ff-47bca2cbba80","Type":"ContainerDied","Data":"25b90357f748056eaf69973176fd2dfa1fcefc48dd5576e9d255770896684796"} Oct 29 23:30:29.683730 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.683710 6507 generic.go:356] "Generic (PLEG): container finished" podID="60ba8a59-d4e6-4d18-ba32-3c6641e543c4" containerID="77168e06a14518aae60f8718992b888a0791e1b37eda1cdb613dbd77fc9588ce" exitCode=0 Oct 29 23:30:29.683842 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.683811 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" event={"ID":"60ba8a59-d4e6-4d18-ba32-3c6641e543c4","Type":"ContainerDied","Data":"77168e06a14518aae60f8718992b888a0791e1b37eda1cdb613dbd77fc9588ce"} Oct 29 23:30:29.689798 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.689744 6507 generic.go:356] "Generic (PLEG): container finished" podID="50c831dd-76a8-4e42-90b4-19847ae771e4" containerID="0ebd4baad9857e2bd2e0580783ff3ee5b2953d22c1239a124f1ff8b08ac10097" exitCode=0 Oct 29 23:30:29.690198 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.689844 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" event={"ID":"50c831dd-76a8-4e42-90b4-19847ae771e4","Type":"ContainerDied","Data":"0ebd4baad9857e2bd2e0580783ff3ee5b2953d22c1239a124f1ff8b08ac10097"} Oct 29 23:30:29.690198 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.689885 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" event={"ID":"50c831dd-76a8-4e42-90b4-19847ae771e4","Type":"ContainerDied","Data":"23f2b98338c2fc18328049fb9ac7004132a0a1b44e1f9e1c2225f7b4f98c799c"} Oct 29 23:30:29.690198 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.689963 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z" Oct 29 23:30:29.693409 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.693387 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" event={"ID":"10f9be75-9161-4cd4-9b75-8428406eed15","Type":"ContainerDied","Data":"7af60fd31a3086b0c72e4236f97d47315628b0d1ee1930463e89b00b725ed257"} Oct 29 23:30:29.693504 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.693409 6507 scope.go:122] "RemoveContainer" containerID="4817da9f0cc3a031e8d9134c698a5ea46c41e9db506f76343d3f318a85e8d444" Oct 29 23:30:29.693504 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.693426 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr" Oct 29 23:30:29.693754 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:29.693734 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4817da9f0cc3a031e8d9134c698a5ea46c41e9db506f76343d3f318a85e8d444\": container with ID starting with 4817da9f0cc3a031e8d9134c698a5ea46c41e9db506f76343d3f318a85e8d444 not found: ID does not exist" containerID="4817da9f0cc3a031e8d9134c698a5ea46c41e9db506f76343d3f318a85e8d444" Oct 29 23:30:29.693803 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.693761 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4817da9f0cc3a031e8d9134c698a5ea46c41e9db506f76343d3f318a85e8d444"} err="failed to get container status \"4817da9f0cc3a031e8d9134c698a5ea46c41e9db506f76343d3f318a85e8d444\": rpc error: code = NotFound desc = could not find container \"4817da9f0cc3a031e8d9134c698a5ea46c41e9db506f76343d3f318a85e8d444\": container with ID starting with 4817da9f0cc3a031e8d9134c698a5ea46c41e9db506f76343d3f318a85e8d444 not found: ID does not exist" Oct 29 23:30:29.693803 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.693778 6507 scope.go:122] "RemoveContainer" containerID="1510a821cfcab096663700e38dae9f8deb17b5da76daa6ff2cb3bc4067d7936e" Oct 29 23:30:29.721845 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.716349 6507 scope.go:122] "RemoveContainer" containerID="1510a821cfcab096663700e38dae9f8deb17b5da76daa6ff2cb3bc4067d7936e" Oct 29 23:30:29.721845 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:29.716641 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1510a821cfcab096663700e38dae9f8deb17b5da76daa6ff2cb3bc4067d7936e\": container with ID starting with 1510a821cfcab096663700e38dae9f8deb17b5da76daa6ff2cb3bc4067d7936e not found: ID does not exist" containerID="1510a821cfcab096663700e38dae9f8deb17b5da76daa6ff2cb3bc4067d7936e" Oct 29 23:30:29.721845 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.716663 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1510a821cfcab096663700e38dae9f8deb17b5da76daa6ff2cb3bc4067d7936e"} err="failed to get container status \"1510a821cfcab096663700e38dae9f8deb17b5da76daa6ff2cb3bc4067d7936e\": rpc error: code = NotFound desc = could not find container \"1510a821cfcab096663700e38dae9f8deb17b5da76daa6ff2cb3bc4067d7936e\": container with ID starting with 1510a821cfcab096663700e38dae9f8deb17b5da76daa6ff2cb3bc4067d7936e not found: ID does not exist" Oct 29 23:30:29.721845 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.716679 6507 scope.go:122] "RemoveContainer" containerID="0ebd4baad9857e2bd2e0580783ff3ee5b2953d22c1239a124f1ff8b08ac10097" Oct 29 23:30:29.727397 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.727346 6507 scope.go:122] "RemoveContainer" containerID="0ebd4baad9857e2bd2e0580783ff3ee5b2953d22c1239a124f1ff8b08ac10097" Oct 29 23:30:29.727906 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:29.727875 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ebd4baad9857e2bd2e0580783ff3ee5b2953d22c1239a124f1ff8b08ac10097\": container with ID starting with 0ebd4baad9857e2bd2e0580783ff3ee5b2953d22c1239a124f1ff8b08ac10097 not found: ID does not exist" containerID="0ebd4baad9857e2bd2e0580783ff3ee5b2953d22c1239a124f1ff8b08ac10097" Oct 29 23:30:29.728055 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.728020 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ebd4baad9857e2bd2e0580783ff3ee5b2953d22c1239a124f1ff8b08ac10097"} err="failed to get container status \"0ebd4baad9857e2bd2e0580783ff3ee5b2953d22c1239a124f1ff8b08ac10097\": rpc error: code = NotFound desc = could not find container \"0ebd4baad9857e2bd2e0580783ff3ee5b2953d22c1239a124f1ff8b08ac10097\": container with ID starting with 0ebd4baad9857e2bd2e0580783ff3ee5b2953d22c1239a124f1ff8b08ac10097 not found: ID does not exist" Oct 29 23:30:29.728164 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.728143 6507 scope.go:122] "RemoveContainer" containerID="3cc034b607740335b7b1c3dbdbdb80110f3640cb10d1bc98d4e6bd3a6a41bda3" Oct 29 23:30:29.751835 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.751786 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr"] Oct 29 23:30:29.758152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.758123 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-837245/premium-w6-837245-kserve-69bd764b57-c7vdr"] Oct 29 23:30:29.784015 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.783990 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr"] Oct 29 23:30:29.789348 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.789317 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-032fa8-68b779d954-ttstr"] Oct 29 23:30:29.817446 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.817409 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k"] Oct 29 23:30:29.821297 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.821269 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-837245/distinct-w6-837245-kserve-545b6454bd-js42k"] Oct 29 23:30:29.894147 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.894115 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z"] Oct 29 23:30:29.902338 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:29.902306 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-032fa8-54cd86599f-v4z8z"] Oct 29 23:30:30.497858 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:30.497816 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" podUID="71038474-e817-4ff1-be48-145e6f10cb3c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:30:30.756515 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:30.756409 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f9be75-9161-4cd4-9b75-8428406eed15" path="/var/lib/kubelet/pods/10f9be75-9161-4cd4-9b75-8428406eed15/volumes" Oct 29 23:30:30.756953 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:30.756892 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" path="/var/lib/kubelet/pods/24364d6f-9a22-4b9c-b2ff-47bca2cbba80/volumes" Oct 29 23:30:30.757249 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:30.757232 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c831dd-76a8-4e42-90b4-19847ae771e4" path="/var/lib/kubelet/pods/50c831dd-76a8-4e42-90b4-19847ae771e4/volumes" Oct 29 23:30:30.757912 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:30.757891 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" path="/var/lib/kubelet/pods/87843995-cc71-42f6-ac3d-a5f3b13907c0/volumes" Oct 29 23:30:30.758291 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:30.758276 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" path="/var/lib/kubelet/pods/bb0c6b55-eae3-4617-a503-ab72e111c5fa/volumes" Oct 29 23:30:30.971984 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:30.971959 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" Oct 29 23:30:31.001528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.001505 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" Oct 29 23:30:31.059654 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.059573 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73c49368-4334-4699-99dc-a1c19dbc9fce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73c49368-4334-4699-99dc-a1c19dbc9fce-maas-api-xks-ca\") pod \"73c49368-4334-4699-99dc-a1c19dbc9fce\" (UID: \"73c49368-4334-4699-99dc-a1c19dbc9fce\") " Oct 29 23:30:31.059815 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.059712 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c49368-4334-4699-99dc-a1c19dbc9fce-kube-api-access-k8j4s\" (UniqueName: \"kubernetes.io/projected/73c49368-4334-4699-99dc-a1c19dbc9fce-kube-api-access-k8j4s\") pod \"73c49368-4334-4699-99dc-a1c19dbc9fce\" (UID: \"73c49368-4334-4699-99dc-a1c19dbc9fce\") " Oct 29 23:30:31.059895 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.059861 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c49368-4334-4699-99dc-a1c19dbc9fce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73c49368-4334-4699-99dc-a1c19dbc9fce-maas-api-openshift-ca\") pod \"73c49368-4334-4699-99dc-a1c19dbc9fce\" (UID: \"73c49368-4334-4699-99dc-a1c19dbc9fce\") " Oct 29 23:30:31.060222 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.060194 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c49368-4334-4699-99dc-a1c19dbc9fce-maas-api-openshift-ca" pod "73c49368-4334-4699-99dc-a1c19dbc9fce" (UID: "73c49368-4334-4699-99dc-a1c19dbc9fce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:31.060453 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.060408 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73c49368-4334-4699-99dc-a1c19dbc9fce-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:31.061951 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.061926 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c49368-4334-4699-99dc-a1c19dbc9fce-maas-api-xks-ca" pod "73c49368-4334-4699-99dc-a1c19dbc9fce" (UID: "73c49368-4334-4699-99dc-a1c19dbc9fce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:30:31.062215 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.062188 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c49368-4334-4699-99dc-a1c19dbc9fce-kube-api-access-k8j4s" pod "73c49368-4334-4699-99dc-a1c19dbc9fce" (UID: "73c49368-4334-4699-99dc-a1c19dbc9fce"). InnerVolumeSpecName "kube-api-access-k8j4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:31.161561 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.161519 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-kube-api-access-72mh5\" (UniqueName: \"kubernetes.io/projected/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-kube-api-access-72mh5\") pod \"60ba8a59-d4e6-4d18-ba32-3c6641e543c4\" (UID: \"60ba8a59-d4e6-4d18-ba32-3c6641e543c4\") " Oct 29 23:30:31.161736 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.161614 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-maas-api-xks-ca\") pod \"60ba8a59-d4e6-4d18-ba32-3c6641e543c4\" (UID: \"60ba8a59-d4e6-4d18-ba32-3c6641e543c4\") " Oct 29 23:30:31.161736 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.161684 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-maas-api-openshift-ca\") pod \"60ba8a59-d4e6-4d18-ba32-3c6641e543c4\" (UID: \"60ba8a59-d4e6-4d18-ba32-3c6641e543c4\") " Oct 29 23:30:31.162079 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.162049 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-maas-api-openshift-ca" pod "60ba8a59-d4e6-4d18-ba32-3c6641e543c4" (UID: "60ba8a59-d4e6-4d18-ba32-3c6641e543c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:31.162300 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.162283 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73c49368-4334-4699-99dc-a1c19dbc9fce-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:31.162350 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.162306 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8j4s\" (UniqueName: \"kubernetes.io/projected/73c49368-4334-4699-99dc-a1c19dbc9fce-kube-api-access-k8j4s\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:31.162350 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.162321 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:31.163693 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.163671 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-kube-api-access-72mh5" pod "60ba8a59-d4e6-4d18-ba32-3c6641e543c4" (UID: "60ba8a59-d4e6-4d18-ba32-3c6641e543c4"). InnerVolumeSpecName "kube-api-access-72mh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:31.163693 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.163686 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-maas-api-xks-ca" pod "60ba8a59-d4e6-4d18-ba32-3c6641e543c4" (UID: "60ba8a59-d4e6-4d18-ba32-3c6641e543c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:30:31.263928 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.263892 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72mh5\" (UniqueName: \"kubernetes.io/projected/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-kube-api-access-72mh5\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:31.263928 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.263924 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ba8a59-d4e6-4d18-ba32-3c6641e543c4-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:31.713944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.713905 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" Oct 29 23:30:31.713944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.713935 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd" event={"ID":"73c49368-4334-4699-99dc-a1c19dbc9fce","Type":"ContainerDied","Data":"3d8dd7279a7cc68910f40d335846cf971b0e31cf9a00d367fdb492a42c821da1"} Oct 29 23:30:31.714149 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.713993 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d8dd7279a7cc68910f40d335846cf971b0e31cf9a00d367fdb492a42c821da1" Oct 29 23:30:31.715337 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.715312 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" event={"ID":"60ba8a59-d4e6-4d18-ba32-3c6641e543c4","Type":"ContainerDied","Data":"b2b19d47b718b2c3044105667ee6562f931e6465f16c6c5db1ccff6a7010c610"} Oct 29 23:30:31.715337 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.715335 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn" Oct 29 23:30:31.715569 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:31.715348 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2b19d47b718b2c3044105667ee6562f931e6465f16c6c5db1ccff6a7010c610" Oct 29 23:30:35.855400 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:35.855355 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft"] Oct 29 23:30:35.855881 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:35.855603 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" podUID="b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" containerName="istio-proxy" containerID="cri-o://55e7a2a10d25f882f13d266d94d1f18ff7b1c7050d4137162f214c2d8db3b566" gracePeriod=30 Oct 29 23:30:35.863828 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:35.863800 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx"] Oct 29 23:30:35.864054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:35.864027 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" podUID="d5a70a55-4e5b-455a-b39f-e6e1b2489d03" containerName="istio-proxy" containerID="cri-o://fa6e180ed393c7afd77b075bfdc4ace3e09a6ed4f5cd3b7c40c1ebb2ff42a52e" gracePeriod=30 Oct 29 23:30:35.873687 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:35.872957 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw"] Oct 29 23:30:35.873687 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:35.873245 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" podUID="150ee23d-c37d-433f-812d-e11da74045c3" containerName="istio-proxy" containerID="cri-o://b1c52ed5f836a3c68b061c1373319be0d2549f0e5a4553f63f58f8e6cde9d646" gracePeriod=30 Oct 29 23:30:36.141535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:36.128229 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:30:39.488389 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:39.488344 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" podUID="cb462872-9968-4e91-96c9-675ffd093457" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:30:39.489141 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:39.488359 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" podUID="56b5ecf5-e109-433c-963d-bf97a5bd6f8a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:30:39.489141 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:39.488361 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" podUID="8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:30:40.496635 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:40.496591 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" podUID="71038474-e817-4ff1-be48-145e6f10cb3c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:30:41.168540 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.168508 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:30:41.202119 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.202086 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istiod-ca-cert\") pod \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " Oct 29 23:30:41.202277 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.202187 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-envoy\") pod \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " Oct 29 23:30:41.202277 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.202259 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-data\") pod \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " Oct 29 23:30:41.202364 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.202299 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-podinfo\") pod \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " Oct 29 23:30:41.202364 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.202339 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-token\" (UniqueName: \"kubernetes.io/projected/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-token\") pod \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " Oct 29 23:30:41.202504 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.202383 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-kube-api-access-52zcl\" (UniqueName: \"kubernetes.io/projected/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-kube-api-access-52zcl\") pod \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " Oct 29 23:30:41.202504 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.202420 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-workload-certs\") pod \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " Oct 29 23:30:41.202504 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.202426 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istiod-ca-cert" pod "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" (UID: "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:41.202504 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.202494 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-credential-socket\") pod \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " Oct 29 23:30:41.202746 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.202556 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-workload-socket\") pod \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\" (UID: \"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17\") " Oct 29 23:30:41.202746 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.202642 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-data" pod "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" (UID: "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:41.202868 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.202766 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:30:41.202868 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.202852 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-workload-certs" pod "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" (UID: "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:41.203089 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.203067 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-workload-socket" pod "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" (UID: "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:41.203089 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.203076 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-credential-socket" pod "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" (UID: "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:41.203301 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.203278 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.203386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.203307 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.203386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.203323 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.203386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.203337 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.203386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.203352 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.204722 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.204697 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-podinfo" pod "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" (UID: "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:30:41.204814 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.204745 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-token" pod "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" (UID: "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:41.204814 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.204804 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-kube-api-access-52zcl" pod "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" (UID: "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17"). InnerVolumeSpecName "kube-api-access-52zcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:41.204984 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.204965 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-envoy" pod "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" (UID: "b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:41.213940 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.213910 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:30:41.304212 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304172 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-istio-envoy\") pod \"150ee23d-c37d-433f-812d-e11da74045c3\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " Oct 29 23:30:41.304423 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304255 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150ee23d-c37d-433f-812d-e11da74045c3-kube-api-access-rzw8j\" (UniqueName: \"kubernetes.io/projected/150ee23d-c37d-433f-812d-e11da74045c3-kube-api-access-rzw8j\") pod \"150ee23d-c37d-433f-812d-e11da74045c3\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " Oct 29 23:30:41.304423 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304295 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-podinfo\") pod \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " Oct 29 23:30:41.304423 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304339 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istiod-ca-cert\") pod \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " Oct 29 23:30:41.304423 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304393 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-workload-certs\") pod \"150ee23d-c37d-433f-812d-e11da74045c3\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " Oct 29 23:30:41.304669 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304434 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-workload-certs\") pod \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " Oct 29 23:30:41.304669 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304474 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-istio-data\") pod \"150ee23d-c37d-433f-812d-e11da74045c3\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " Oct 29 23:30:41.304669 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304504 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-credential-socket\") pod \"150ee23d-c37d-433f-812d-e11da74045c3\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " Oct 29 23:30:41.304669 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304549 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-envoy\") pod \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " Oct 29 23:30:41.304669 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304631 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-workload-socket\") pod \"150ee23d-c37d-433f-812d-e11da74045c3\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " Oct 29 23:30:41.304930 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304712 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-credential-socket\") pod \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " Oct 29 23:30:41.304930 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304738 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-token\" (UniqueName: \"kubernetes.io/projected/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-token\") pod \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " Oct 29 23:30:41.304930 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304768 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150ee23d-c37d-433f-812d-e11da74045c3-istio-token\" (UniqueName: \"kubernetes.io/projected/150ee23d-c37d-433f-812d-e11da74045c3-istio-token\") pod \"150ee23d-c37d-433f-812d-e11da74045c3\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " Oct 29 23:30:41.304930 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304806 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/150ee23d-c37d-433f-812d-e11da74045c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150ee23d-c37d-433f-812d-e11da74045c3-istiod-ca-cert\") pod \"150ee23d-c37d-433f-812d-e11da74045c3\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " Oct 29 23:30:41.304930 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304849 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-kube-api-access-7gxlx\" (UniqueName: \"kubernetes.io/projected/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-kube-api-access-7gxlx\") pod \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " Oct 29 23:30:41.304930 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304919 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-data\") pod \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " Oct 29 23:30:41.305235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.304959 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/150ee23d-c37d-433f-812d-e11da74045c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150ee23d-c37d-433f-812d-e11da74045c3-istio-podinfo\") pod \"150ee23d-c37d-433f-812d-e11da74045c3\" (UID: \"150ee23d-c37d-433f-812d-e11da74045c3\") " Oct 29 23:30:41.305235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.305061 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-workload-socket\") pod \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\" (UID: \"d5a70a55-4e5b-455a-b39f-e6e1b2489d03\") " Oct 29 23:30:41.305743 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.305696 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52zcl\" (UniqueName: \"kubernetes.io/projected/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-kube-api-access-52zcl\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.305743 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.305729 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.305743 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.305744 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.306037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.305757 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.306037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.305974 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-workload-socket" pod "d5a70a55-4e5b-455a-b39f-e6e1b2489d03" (UID: "d5a70a55-4e5b-455a-b39f-e6e1b2489d03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:41.306215 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.306189 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-workload-socket" pod "150ee23d-c37d-433f-812d-e11da74045c3" (UID: "150ee23d-c37d-433f-812d-e11da74045c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:41.306515 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.306419 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-credential-socket" pod "d5a70a55-4e5b-455a-b39f-e6e1b2489d03" (UID: "d5a70a55-4e5b-455a-b39f-e6e1b2489d03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:41.307066 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.307039 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istiod-ca-cert" pod "d5a70a55-4e5b-455a-b39f-e6e1b2489d03" (UID: "d5a70a55-4e5b-455a-b39f-e6e1b2489d03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:41.307914 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.307758 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-credential-socket" pod "150ee23d-c37d-433f-812d-e11da74045c3" (UID: "150ee23d-c37d-433f-812d-e11da74045c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:41.307914 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.307891 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-envoy" pod "d5a70a55-4e5b-455a-b39f-e6e1b2489d03" (UID: "d5a70a55-4e5b-455a-b39f-e6e1b2489d03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:41.308396 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.308124 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-istio-data" pod "150ee23d-c37d-433f-812d-e11da74045c3" (UID: "150ee23d-c37d-433f-812d-e11da74045c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:41.308839 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.308551 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-workload-certs" pod "150ee23d-c37d-433f-812d-e11da74045c3" (UID: "150ee23d-c37d-433f-812d-e11da74045c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:41.309444 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.309330 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-token" pod "d5a70a55-4e5b-455a-b39f-e6e1b2489d03" (UID: "d5a70a55-4e5b-455a-b39f-e6e1b2489d03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:41.309923 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.309682 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-data" pod "d5a70a55-4e5b-455a-b39f-e6e1b2489d03" (UID: "d5a70a55-4e5b-455a-b39f-e6e1b2489d03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:41.310185 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.310144 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150ee23d-c37d-433f-812d-e11da74045c3-istiod-ca-cert" pod "150ee23d-c37d-433f-812d-e11da74045c3" (UID: "150ee23d-c37d-433f-812d-e11da74045c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:41.310275 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.310221 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150ee23d-c37d-433f-812d-e11da74045c3-kube-api-access-rzw8j" pod "150ee23d-c37d-433f-812d-e11da74045c3" (UID: "150ee23d-c37d-433f-812d-e11da74045c3"). InnerVolumeSpecName "kube-api-access-rzw8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:41.310666 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.310643 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-workload-certs" pod "d5a70a55-4e5b-455a-b39f-e6e1b2489d03" (UID: "d5a70a55-4e5b-455a-b39f-e6e1b2489d03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:41.311207 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.311186 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-istio-envoy" pod "150ee23d-c37d-433f-812d-e11da74045c3" (UID: "150ee23d-c37d-433f-812d-e11da74045c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:41.311687 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.311659 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150ee23d-c37d-433f-812d-e11da74045c3-istio-token" pod "150ee23d-c37d-433f-812d-e11da74045c3" (UID: "150ee23d-c37d-433f-812d-e11da74045c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:41.312539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.312513 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-kube-api-access-7gxlx" pod "d5a70a55-4e5b-455a-b39f-e6e1b2489d03" (UID: "d5a70a55-4e5b-455a-b39f-e6e1b2489d03"). InnerVolumeSpecName "kube-api-access-7gxlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:41.312634 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.312560 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/150ee23d-c37d-433f-812d-e11da74045c3-istio-podinfo" pod "150ee23d-c37d-433f-812d-e11da74045c3" (UID: "150ee23d-c37d-433f-812d-e11da74045c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:30:41.313062 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.313042 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-podinfo" pod "d5a70a55-4e5b-455a-b39f-e6e1b2489d03" (UID: "d5a70a55-4e5b-455a-b39f-e6e1b2489d03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:30:41.407529 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407434 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407529 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407499 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407529 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407514 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzw8j\" (UniqueName: \"kubernetes.io/projected/150ee23d-c37d-433f-812d-e11da74045c3-kube-api-access-rzw8j\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407529 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407527 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407541 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407554 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407565 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407575 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407588 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407599 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407609 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150ee23d-c37d-433f-812d-e11da74045c3-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407623 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407637 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407649 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150ee23d-c37d-433f-812d-e11da74045c3-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407662 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150ee23d-c37d-433f-812d-e11da74045c3-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407673 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gxlx\" (UniqueName: \"kubernetes.io/projected/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-kube-api-access-7gxlx\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407685 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a70a55-4e5b-455a-b39f-e6e1b2489d03-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.407807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.407696 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150ee23d-c37d-433f-812d-e11da74045c3-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:41.952245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.952206 6507 generic.go:356] "Generic (PLEG): container finished" podID="d5a70a55-4e5b-455a-b39f-e6e1b2489d03" containerID="fa6e180ed393c7afd77b075bfdc4ace3e09a6ed4f5cd3b7c40c1ebb2ff42a52e" exitCode=0 Oct 29 23:30:41.952942 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.952873 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" Oct 29 23:30:41.953316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.953207 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" event={"ID":"d5a70a55-4e5b-455a-b39f-e6e1b2489d03","Type":"ContainerDied","Data":"fa6e180ed393c7afd77b075bfdc4ace3e09a6ed4f5cd3b7c40c1ebb2ff42a52e"} Oct 29 23:30:41.953550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.953525 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx" event={"ID":"d5a70a55-4e5b-455a-b39f-e6e1b2489d03","Type":"ContainerDied","Data":"102a04eb82cd0acce69d737adc142b335c731cae8e763affbe40b60cb6087900"} Oct 29 23:30:41.953688 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.953674 6507 scope.go:122] "RemoveContainer" containerID="fa6e180ed393c7afd77b075bfdc4ace3e09a6ed4f5cd3b7c40c1ebb2ff42a52e" Oct 29 23:30:41.974490 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.969723 6507 generic.go:356] "Generic (PLEG): container finished" podID="150ee23d-c37d-433f-812d-e11da74045c3" containerID="b1c52ed5f836a3c68b061c1373319be0d2549f0e5a4553f63f58f8e6cde9d646" exitCode=0 Oct 29 23:30:41.974490 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.969893 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" event={"ID":"150ee23d-c37d-433f-812d-e11da74045c3","Type":"ContainerDied","Data":"b1c52ed5f836a3c68b061c1373319be0d2549f0e5a4553f63f58f8e6cde9d646"} Oct 29 23:30:41.974490 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.969947 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" event={"ID":"150ee23d-c37d-433f-812d-e11da74045c3","Type":"ContainerDied","Data":"9df94f84fa6eec1886492866983e222b9429d3310940717e7543c30f72bd35c8"} Oct 29 23:30:41.974490 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.970031 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw" Oct 29 23:30:41.984065 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.984035 6507 generic.go:356] "Generic (PLEG): container finished" podID="b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" containerID="55e7a2a10d25f882f13d266d94d1f18ff7b1c7050d4137162f214c2d8db3b566" exitCode=0 Oct 29 23:30:41.984213 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.984114 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" Oct 29 23:30:41.984213 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.984161 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" event={"ID":"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17","Type":"ContainerDied","Data":"55e7a2a10d25f882f13d266d94d1f18ff7b1c7050d4137162f214c2d8db3b566"} Oct 29 23:30:41.984213 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.984204 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft" event={"ID":"b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17","Type":"ContainerDied","Data":"944fa190e6eb11c3ddc6d4ca0474b1b9ec42956763ac896ddd5df48f24378bca"} Oct 29 23:30:41.994310 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.994128 6507 scope.go:122] "RemoveContainer" containerID="fa6e180ed393c7afd77b075bfdc4ace3e09a6ed4f5cd3b7c40c1ebb2ff42a52e" Oct 29 23:30:41.994555 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:41.994449 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6e180ed393c7afd77b075bfdc4ace3e09a6ed4f5cd3b7c40c1ebb2ff42a52e\": container with ID starting with fa6e180ed393c7afd77b075bfdc4ace3e09a6ed4f5cd3b7c40c1ebb2ff42a52e not found: ID does not exist" containerID="fa6e180ed393c7afd77b075bfdc4ace3e09a6ed4f5cd3b7c40c1ebb2ff42a52e" Oct 29 23:30:41.994628 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.994565 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6e180ed393c7afd77b075bfdc4ace3e09a6ed4f5cd3b7c40c1ebb2ff42a52e"} err="failed to get container status \"fa6e180ed393c7afd77b075bfdc4ace3e09a6ed4f5cd3b7c40c1ebb2ff42a52e\": rpc error: code = NotFound desc = could not find container \"fa6e180ed393c7afd77b075bfdc4ace3e09a6ed4f5cd3b7c40c1ebb2ff42a52e\": container with ID starting with fa6e180ed393c7afd77b075bfdc4ace3e09a6ed4f5cd3b7c40c1ebb2ff42a52e not found: ID does not exist" Oct 29 23:30:41.994628 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:41.994586 6507 scope.go:122] "RemoveContainer" containerID="b1c52ed5f836a3c68b061c1373319be0d2549f0e5a4553f63f58f8e6cde9d646" Oct 29 23:30:42.009136 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:42.009108 6507 scope.go:122] "RemoveContainer" containerID="b1c52ed5f836a3c68b061c1373319be0d2549f0e5a4553f63f58f8e6cde9d646" Oct 29 23:30:42.009566 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:42.009450 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c52ed5f836a3c68b061c1373319be0d2549f0e5a4553f63f58f8e6cde9d646\": container with ID starting with b1c52ed5f836a3c68b061c1373319be0d2549f0e5a4553f63f58f8e6cde9d646 not found: ID does not exist" containerID="b1c52ed5f836a3c68b061c1373319be0d2549f0e5a4553f63f58f8e6cde9d646" Oct 29 23:30:42.009628 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:42.009580 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c52ed5f836a3c68b061c1373319be0d2549f0e5a4553f63f58f8e6cde9d646"} err="failed to get container status \"b1c52ed5f836a3c68b061c1373319be0d2549f0e5a4553f63f58f8e6cde9d646\": rpc error: code = NotFound desc = could not find container \"b1c52ed5f836a3c68b061c1373319be0d2549f0e5a4553f63f58f8e6cde9d646\": container with ID starting with b1c52ed5f836a3c68b061c1373319be0d2549f0e5a4553f63f58f8e6cde9d646 not found: ID does not exist" Oct 29 23:30:42.009628 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:42.009603 6507 scope.go:122] "RemoveContainer" containerID="55e7a2a10d25f882f13d266d94d1f18ff7b1c7050d4137162f214c2d8db3b566" Oct 29 23:30:42.020232 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:42.020206 6507 scope.go:122] "RemoveContainer" containerID="55e7a2a10d25f882f13d266d94d1f18ff7b1c7050d4137162f214c2d8db3b566" Oct 29 23:30:42.020582 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:42.020562 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e7a2a10d25f882f13d266d94d1f18ff7b1c7050d4137162f214c2d8db3b566\": container with ID starting with 55e7a2a10d25f882f13d266d94d1f18ff7b1c7050d4137162f214c2d8db3b566 not found: ID does not exist" containerID="55e7a2a10d25f882f13d266d94d1f18ff7b1c7050d4137162f214c2d8db3b566" Oct 29 23:30:42.020648 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:42.020592 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e7a2a10d25f882f13d266d94d1f18ff7b1c7050d4137162f214c2d8db3b566"} err="failed to get container status \"55e7a2a10d25f882f13d266d94d1f18ff7b1c7050d4137162f214c2d8db3b566\": rpc error: code = NotFound desc = could not find container \"55e7a2a10d25f882f13d266d94d1f18ff7b1c7050d4137162f214c2d8db3b566\": container with ID starting with 55e7a2a10d25f882f13d266d94d1f18ff7b1c7050d4137162f214c2d8db3b566 not found: ID does not exist" Oct 29 23:30:42.033101 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:42.033072 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw"] Oct 29 23:30:42.041513 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:42.041485 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-x2hdw"] Oct 29 23:30:42.072777 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:42.072740 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx"] Oct 29 23:30:42.080887 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:42.080855 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-gcvhx"] Oct 29 23:30:42.106241 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:42.106202 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft"] Oct 29 23:30:42.112368 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:42.112341 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-032fa8-openshift-default-7b6d48f8fc-xzkft"] Oct 29 23:30:42.753165 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:42.753134 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="150ee23d-c37d-433f-812d-e11da74045c3" path="/var/lib/kubelet/pods/150ee23d-c37d-433f-812d-e11da74045c3/volumes" Oct 29 23:30:42.753643 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:42.753629 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" path="/var/lib/kubelet/pods/b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17/volumes" Oct 29 23:30:42.754065 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:42.754053 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a70a55-4e5b-455a-b39f-e6e1b2489d03" path="/var/lib/kubelet/pods/d5a70a55-4e5b-455a-b39f-e6e1b2489d03/volumes" Oct 29 23:30:43.779735 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:43.779707 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" Oct 29 23:30:43.802181 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:43.802152 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" Oct 29 23:30:43.836884 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:43.836850 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26-plugins-config-volume\") pod \"8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26\" (UID: \"8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26\") " Oct 29 23:30:43.837065 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:43.836908 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26-kube-api-access-4czb8\" (UniqueName: \"kubernetes.io/projected/8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26-kube-api-access-4czb8\") pod \"8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26\" (UID: \"8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26\") " Oct 29 23:30:43.837130 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:43.837069 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b5ecf5-e109-433c-963d-bf97a5bd6f8a-kube-api-access-kct6j\" (UniqueName: \"kubernetes.io/projected/56b5ecf5-e109-433c-963d-bf97a5bd6f8a-kube-api-access-kct6j\") pod \"56b5ecf5-e109-433c-963d-bf97a5bd6f8a\" (UID: \"56b5ecf5-e109-433c-963d-bf97a5bd6f8a\") " Oct 29 23:30:43.837189 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:43.837166 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56b5ecf5-e109-433c-963d-bf97a5bd6f8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b5ecf5-e109-433c-963d-bf97a5bd6f8a-plugins-config-volume\") pod \"56b5ecf5-e109-433c-963d-bf97a5bd6f8a\" (UID: \"56b5ecf5-e109-433c-963d-bf97a5bd6f8a\") " Oct 29 23:30:43.837326 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:43.837296 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26-plugins-config-volume" pod "8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26" (UID: "8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:43.837623 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:43.837597 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b5ecf5-e109-433c-963d-bf97a5bd6f8a-plugins-config-volume" pod "56b5ecf5-e109-433c-963d-bf97a5bd6f8a" (UID: "56b5ecf5-e109-433c-963d-bf97a5bd6f8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:43.838064 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:43.838038 6507 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b5ecf5-e109-433c-963d-bf97a5bd6f8a-plugins-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:43.838176 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:43.838067 6507 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26-plugins-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:43.839331 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:43.839305 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b5ecf5-e109-433c-963d-bf97a5bd6f8a-kube-api-access-kct6j" pod "56b5ecf5-e109-433c-963d-bf97a5bd6f8a" (UID: "56b5ecf5-e109-433c-963d-bf97a5bd6f8a"). InnerVolumeSpecName "kube-api-access-kct6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:43.839447 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:43.839407 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26-kube-api-access-4czb8" pod "8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26" (UID: "8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26"). InnerVolumeSpecName "kube-api-access-4czb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:43.939053 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:43.939019 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kct6j\" (UniqueName: \"kubernetes.io/projected/56b5ecf5-e109-433c-963d-bf97a5bd6f8a-kube-api-access-kct6j\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:43.939053 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:43.939048 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4czb8\" (UniqueName: \"kubernetes.io/projected/8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26-kube-api-access-4czb8\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:44.003012 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.002915 6507 generic.go:356] "Generic (PLEG): container finished" podID="8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26" containerID="90da7b065cc3fdb3456ce9da4ca0713123a51c97766fc4b481522497c07c9d05" exitCode=0 Oct 29 23:30:44.003012 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.002968 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" Oct 29 23:30:44.003012 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.002998 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" event={"ID":"8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26","Type":"ContainerDied","Data":"90da7b065cc3fdb3456ce9da4ca0713123a51c97766fc4b481522497c07c9d05"} Oct 29 23:30:44.003269 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.003048 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2" event={"ID":"8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26","Type":"ContainerDied","Data":"0c54abe784dcc295efbe88f0c210e1a082f2fba155ba5d3319e159a0e85258d9"} Oct 29 23:30:44.003269 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.003069 6507 scope.go:122] "RemoveContainer" containerID="90da7b065cc3fdb3456ce9da4ca0713123a51c97766fc4b481522497c07c9d05" Oct 29 23:30:44.004521 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.004494 6507 generic.go:356] "Generic (PLEG): container finished" podID="56b5ecf5-e109-433c-963d-bf97a5bd6f8a" containerID="cf43714541cc6be5c9272430fee0b8b1f373d7038cd504c588f67aad97f3d52a" exitCode=0 Oct 29 23:30:44.004625 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.004556 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" Oct 29 23:30:44.004625 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.004577 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" event={"ID":"56b5ecf5-e109-433c-963d-bf97a5bd6f8a","Type":"ContainerDied","Data":"cf43714541cc6be5c9272430fee0b8b1f373d7038cd504c588f67aad97f3d52a"} Oct 29 23:30:44.004727 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.004634 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2" event={"ID":"56b5ecf5-e109-433c-963d-bf97a5bd6f8a","Type":"ContainerDied","Data":"c4e9779b199c3b6aaadd150295da46c403e068354e6a33860c35a8e6b22662a1"} Oct 29 23:30:44.013491 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.013437 6507 scope.go:122] "RemoveContainer" containerID="90da7b065cc3fdb3456ce9da4ca0713123a51c97766fc4b481522497c07c9d05" Oct 29 23:30:44.013844 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:44.013819 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90da7b065cc3fdb3456ce9da4ca0713123a51c97766fc4b481522497c07c9d05\": container with ID starting with 90da7b065cc3fdb3456ce9da4ca0713123a51c97766fc4b481522497c07c9d05 not found: ID does not exist" containerID="90da7b065cc3fdb3456ce9da4ca0713123a51c97766fc4b481522497c07c9d05" Oct 29 23:30:44.013961 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.013848 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90da7b065cc3fdb3456ce9da4ca0713123a51c97766fc4b481522497c07c9d05"} err="failed to get container status \"90da7b065cc3fdb3456ce9da4ca0713123a51c97766fc4b481522497c07c9d05\": rpc error: code = NotFound desc = could not find container \"90da7b065cc3fdb3456ce9da4ca0713123a51c97766fc4b481522497c07c9d05\": container with ID starting with 90da7b065cc3fdb3456ce9da4ca0713123a51c97766fc4b481522497c07c9d05 not found: ID does not exist" Oct 29 23:30:44.013961 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.013866 6507 scope.go:122] "RemoveContainer" containerID="cf43714541cc6be5c9272430fee0b8b1f373d7038cd504c588f67aad97f3d52a" Oct 29 23:30:44.024330 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.024309 6507 scope.go:122] "RemoveContainer" containerID="cf43714541cc6be5c9272430fee0b8b1f373d7038cd504c588f67aad97f3d52a" Oct 29 23:30:44.024714 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:44.024689 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf43714541cc6be5c9272430fee0b8b1f373d7038cd504c588f67aad97f3d52a\": container with ID starting with cf43714541cc6be5c9272430fee0b8b1f373d7038cd504c588f67aad97f3d52a not found: ID does not exist" containerID="cf43714541cc6be5c9272430fee0b8b1f373d7038cd504c588f67aad97f3d52a" Oct 29 23:30:44.024803 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.024720 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf43714541cc6be5c9272430fee0b8b1f373d7038cd504c588f67aad97f3d52a"} err="failed to get container status \"cf43714541cc6be5c9272430fee0b8b1f373d7038cd504c588f67aad97f3d52a\": rpc error: code = NotFound desc = could not find container \"cf43714541cc6be5c9272430fee0b8b1f373d7038cd504c588f67aad97f3d52a\": container with ID starting with cf43714541cc6be5c9272430fee0b8b1f373d7038cd504c588f67aad97f3d52a not found: ID does not exist" Oct 29 23:30:44.058922 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.058878 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2"] Oct 29 23:30:44.062607 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.062566 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2670a5-56f447f6d7-h4df2"] Oct 29 23:30:44.088471 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.088424 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2"] Oct 29 23:30:44.093411 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:44.093380 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-2670a5-558844c756-hn9j2"] Oct 29 23:30:45.019620 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.019512 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56b5ecf5-e109-433c-963d-bf97a5bd6f8a" path="/var/lib/kubelet/pods/56b5ecf5-e109-433c-963d-bf97a5bd6f8a/volumes" Oct 29 23:30:45.074481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.070890 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26" path="/var/lib/kubelet/pods/8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26/volumes" Oct 29 23:30:45.074481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.073213 6507 generic.go:356] "Generic (PLEG): container finished" podID="cb462872-9968-4e91-96c9-675ffd093457" containerID="750047c8bae361f192f8e2f5e12aec9a9bb1fff05002b40f98186bf16b0fb282" exitCode=0 Oct 29 23:30:45.074481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.073303 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" event={"ID":"cb462872-9968-4e91-96c9-675ffd093457","Type":"ContainerDied","Data":"750047c8bae361f192f8e2f5e12aec9a9bb1fff05002b40f98186bf16b0fb282"} Oct 29 23:30:45.080480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.076700 6507 generic.go:356] "Generic (PLEG): container finished" podID="71038474-e817-4ff1-be48-145e6f10cb3c" containerID="b23794173ebc514c6f2e63ad497e1aa15c01986f8039a44e2e63961c5398b5cd" exitCode=0 Oct 29 23:30:45.080480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.076774 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" event={"ID":"71038474-e817-4ff1-be48-145e6f10cb3c","Type":"ContainerDied","Data":"b23794173ebc514c6f2e63ad497e1aa15c01986f8039a44e2e63961c5398b5cd"} Oct 29 23:30:45.586479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.575127 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" Oct 29 23:30:45.650485 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:45.650416 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:30:45.656712 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.656686 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" Oct 29 23:30:45.667016 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.661859 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71038474-e817-4ff1-be48-145e6f10cb3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71038474-e817-4ff1-be48-145e6f10cb3c-plugins-config-volume\") pod \"71038474-e817-4ff1-be48-145e6f10cb3c\" (UID: \"71038474-e817-4ff1-be48-145e6f10cb3c\") " Oct 29 23:30:45.667016 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.663184 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71038474-e817-4ff1-be48-145e6f10cb3c-kube-api-access-llqkr\" (UniqueName: \"kubernetes.io/projected/71038474-e817-4ff1-be48-145e6f10cb3c-kube-api-access-llqkr\") pod \"71038474-e817-4ff1-be48-145e6f10cb3c\" (UID: \"71038474-e817-4ff1-be48-145e6f10cb3c\") " Oct 29 23:30:45.667016 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.666952 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71038474-e817-4ff1-be48-145e6f10cb3c-plugins-config-volume" pod "71038474-e817-4ff1-be48-145e6f10cb3c" (UID: "71038474-e817-4ff1-be48-145e6f10cb3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:45.674092 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.674051 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71038474-e817-4ff1-be48-145e6f10cb3c-kube-api-access-llqkr" pod "71038474-e817-4ff1-be48-145e6f10cb3c" (UID: "71038474-e817-4ff1-be48-145e6f10cb3c"). InnerVolumeSpecName "kube-api-access-llqkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:45.765619 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.765528 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb462872-9968-4e91-96c9-675ffd093457-kube-api-access-52x77\" (UniqueName: \"kubernetes.io/projected/cb462872-9968-4e91-96c9-675ffd093457-kube-api-access-52x77\") pod \"cb462872-9968-4e91-96c9-675ffd093457\" (UID: \"cb462872-9968-4e91-96c9-675ffd093457\") " Oct 29 23:30:45.765789 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.765731 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb462872-9968-4e91-96c9-675ffd093457-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb462872-9968-4e91-96c9-675ffd093457-plugins-config-volume\") pod \"cb462872-9968-4e91-96c9-675ffd093457\" (UID: \"cb462872-9968-4e91-96c9-675ffd093457\") " Oct 29 23:30:45.766020 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.766002 6507 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71038474-e817-4ff1-be48-145e6f10cb3c-plugins-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:45.766020 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.766019 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llqkr\" (UniqueName: \"kubernetes.io/projected/71038474-e817-4ff1-be48-145e6f10cb3c-kube-api-access-llqkr\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:45.766129 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.766085 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb462872-9968-4e91-96c9-675ffd093457-plugins-config-volume" pod "cb462872-9968-4e91-96c9-675ffd093457" (UID: "cb462872-9968-4e91-96c9-675ffd093457"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:45.767791 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.767767 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb462872-9968-4e91-96c9-675ffd093457-kube-api-access-52x77" pod "cb462872-9968-4e91-96c9-675ffd093457" (UID: "cb462872-9968-4e91-96c9-675ffd093457"). InnerVolumeSpecName "kube-api-access-52x77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:45.873511 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.871688 6507 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb462872-9968-4e91-96c9-675ffd093457-plugins-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:45.873511 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:45.871732 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52x77\" (UniqueName: \"kubernetes.io/projected/cb462872-9968-4e91-96c9-675ffd093457-kube-api-access-52x77\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:46.085286 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:46.085198 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" Oct 29 23:30:46.085286 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:46.085211 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p" event={"ID":"cb462872-9968-4e91-96c9-675ffd093457","Type":"ContainerDied","Data":"e2c4416dcc115b47d1d2845762eb3714d09066627a0d17939981b8b78bbdc8a9"} Oct 29 23:30:46.085286 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:46.085279 6507 scope.go:122] "RemoveContainer" containerID="750047c8bae361f192f8e2f5e12aec9a9bb1fff05002b40f98186bf16b0fb282" Oct 29 23:30:46.088496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:46.086886 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" Oct 29 23:30:46.088496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:46.086927 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg" event={"ID":"71038474-e817-4ff1-be48-145e6f10cb3c","Type":"ContainerDied","Data":"8a89a822c6905331746224a9bc7361a7a36c79b73316d9cff32cb94cf5caa5c1"} Oct 29 23:30:46.095726 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:46.095710 6507 scope.go:122] "RemoveContainer" containerID="b23794173ebc514c6f2e63ad497e1aa15c01986f8039a44e2e63961c5398b5cd" Oct 29 23:30:46.131105 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:46.131067 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p"] Oct 29 23:30:46.139982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:46.139953 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-837245-779cb9445f-4wg8p"] Oct 29 23:30:46.160563 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:46.160531 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg"] Oct 29 23:30:46.167722 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:46.167692 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-837245-f69cb8df6-b24gg"] Oct 29 23:30:46.760569 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:46.760450 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71038474-e817-4ff1-be48-145e6f10cb3c" path="/var/lib/kubelet/pods/71038474-e817-4ff1-be48-145e6f10cb3c/volumes" Oct 29 23:30:46.761006 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:46.760985 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb462872-9968-4e91-96c9-675ffd093457" path="/var/lib/kubelet/pods/cb462872-9968-4e91-96c9-675ffd093457/volumes" Oct 29 23:30:53.797481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:53.796549 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf"] Oct 29 23:30:53.797481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:53.796786 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" containerName="istio-proxy" containerID="cri-o://efd22a131c4c954c080e1843e1f49b942f1233c71b0775d5ab24bb334534c242" gracePeriod=30 Oct 29 23:30:53.929501 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:53.882536 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4"] Oct 29 23:30:53.929501 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:53.882871 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" podUID="d02ada27-df84-4ea9-988d-3e3f0516f70a" containerName="istio-proxy" containerID="cri-o://c659e5a2006be61e7e90307624e50779924c7292f29de693a8c0fc0adafae3c3" gracePeriod=30 Oct 29 23:30:53.929501 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:53.892667 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr"] Oct 29 23:30:53.929501 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:53.892909 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" podUID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" containerName="istio-proxy" containerID="cri-o://7e0c75458f20a828c3ebf7997e6f3845f0256d76897ad557ee7a9b95aa5ca6d0" gracePeriod=30 Oct 29 23:30:53.957242 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:53.956740 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6"] Oct 29 23:30:53.957242 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:53.957018 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" podUID="039b4caf-d11b-4933-af15-2efb408c36e3" containerName="istio-proxy" containerID="cri-o://7633ea2be2aeb4c2d990f2afefdc49a55803208dded89996b6db4ae035bdb930" gracePeriod=30 Oct 29 23:30:54.599449 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:54.599409 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5"] Oct 29 23:30:54.599721 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:54.599695 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" podUID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" containerName="istio-proxy" containerID="cri-o://af8b4d7dcd6f28d79776ff315ddc2e0678c21bb4d91d4cd2d4f6fcd34ab88227" gracePeriod=30 Oct 29 23:30:54.608344 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:54.608308 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4"] Oct 29 23:30:54.608621 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:54.608588 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" podUID="aa280316-fb6c-40db-8658-65b84bb4382b" containerName="istio-proxy" containerID="cri-o://d6a25edbc79d32ff0fdc5947f6f45dfd61e520e49b93b29de043754e00bab8bd" gracePeriod=30 Oct 29 23:30:54.625164 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:54.625130 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2"] Oct 29 23:30:54.625402 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:54.625379 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" podUID="12f4ed99-042f-4e41-8489-9af7622e632c" containerName="istio-proxy" containerID="cri-o://2a6cecff018d66364ad7f5d48d752ab3eb2f23b50f5814c81aa66387bfa2ee53" gracePeriod=30 Oct 29 23:30:55.198789 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.198751 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h"] Oct 29 23:30:55.199303 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199287 6507 cpu_manager.go:414] "containerMap: removing container" podUID="b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" containerName="istio-proxy" Oct 29 23:30:55.199345 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199304 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" containerName="istio-proxy" Oct 29 23:30:55.199345 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199320 6507 cpu_manager.go:414] "containerMap: removing container" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" Oct 29 23:30:55.199345 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199325 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" Oct 29 23:30:55.199345 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199338 6507 cpu_manager.go:414] "containerMap: removing container" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" Oct 29 23:30:55.199345 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199343 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199354 6507 cpu_manager.go:414] "containerMap: removing container" podUID="56b5ecf5-e109-433c-963d-bf97a5bd6f8a" containerName="payload-processing" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199359 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b5ecf5-e109-433c-963d-bf97a5bd6f8a" containerName="payload-processing" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199366 6507 cpu_manager.go:414] "containerMap: removing container" podUID="10f9be75-9161-4cd4-9b75-8428406eed15" containerName="payload-processing" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199371 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f9be75-9161-4cd4-9b75-8428406eed15" containerName="payload-processing" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199378 6507 cpu_manager.go:414] "containerMap: removing container" podUID="150ee23d-c37d-433f-812d-e11da74045c3" containerName="istio-proxy" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199382 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150ee23d-c37d-433f-812d-e11da74045c3" containerName="istio-proxy" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199392 6507 cpu_manager.go:414] "containerMap: removing container" podUID="cb462872-9968-4e91-96c9-675ffd093457" containerName="payload-pre-processing" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199396 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb462872-9968-4e91-96c9-675ffd093457" containerName="payload-pre-processing" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199403 6507 cpu_manager.go:414] "containerMap: removing container" podUID="60ba8a59-d4e6-4d18-ba32-3c6641e543c4" containerName="revoke-keys" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199408 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ba8a59-d4e6-4d18-ba32-3c6641e543c4" containerName="revoke-keys" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199415 6507 cpu_manager.go:414] "containerMap: removing container" podUID="8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26" containerName="payload-pre-processing" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199420 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26" containerName="payload-pre-processing" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199429 6507 cpu_manager.go:414] "containerMap: removing container" podUID="50c831dd-76a8-4e42-90b4-19847ae771e4" containerName="payload-pre-processing" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199433 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c831dd-76a8-4e42-90b4-19847ae771e4" containerName="payload-pre-processing" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199444 6507 cpu_manager.go:414] "containerMap: removing container" podUID="d5a70a55-4e5b-455a-b39f-e6e1b2489d03" containerName="istio-proxy" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199448 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a70a55-4e5b-455a-b39f-e6e1b2489d03" containerName="istio-proxy" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199468 6507 cpu_manager.go:414] "containerMap: removing container" podUID="73c49368-4334-4699-99dc-a1c19dbc9fce" containerName="revoke-keys" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199476 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c49368-4334-4699-99dc-a1c19dbc9fce" containerName="revoke-keys" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199493 6507 cpu_manager.go:414] "containerMap: removing container" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199501 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199521 6507 cpu_manager.go:414] "containerMap: removing container" podUID="71038474-e817-4ff1-be48-145e6f10cb3c" containerName="payload-processing" Oct 29 23:30:55.199539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199528 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71038474-e817-4ff1-be48-145e6f10cb3c" containerName="payload-processing" Oct 29 23:30:55.200132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199616 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="24364d6f-9a22-4b9c-b2ff-47bca2cbba80" containerName="main" Oct 29 23:30:55.200132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199627 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b5ecf5-e109-433c-963d-bf97a5bd6f8a" containerName="payload-processing" Oct 29 23:30:55.200132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199632 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f9be75-9161-4cd4-9b75-8428406eed15" containerName="payload-processing" Oct 29 23:30:55.200132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199638 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="150ee23d-c37d-433f-812d-e11da74045c3" containerName="istio-proxy" Oct 29 23:30:55.200132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199644 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb462872-9968-4e91-96c9-675ffd093457" containerName="payload-pre-processing" Oct 29 23:30:55.200132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199649 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ba8a59-d4e6-4d18-ba32-3c6641e543c4" containerName="revoke-keys" Oct 29 23:30:55.200132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199653 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb1736b-d3a6-4de0-bfb5-4a9ce2d90c26" containerName="payload-pre-processing" Oct 29 23:30:55.200132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199659 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c831dd-76a8-4e42-90b4-19847ae771e4" containerName="payload-pre-processing" Oct 29 23:30:55.200132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199665 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a70a55-4e5b-455a-b39f-e6e1b2489d03" containerName="istio-proxy" Oct 29 23:30:55.200132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199669 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c49368-4334-4699-99dc-a1c19dbc9fce" containerName="revoke-keys" Oct 29 23:30:55.200132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199675 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb0c6b55-eae3-4617-a503-ab72e111c5fa" containerName="main" Oct 29 23:30:55.200132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199682 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="71038474-e817-4ff1-be48-145e6f10cb3c" containerName="payload-processing" Oct 29 23:30:55.200132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199689 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ea7446-b9e3-4bc7-a338-0aaaf19e7f17" containerName="istio-proxy" Oct 29 23:30:55.200132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.199696 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="87843995-cc71-42f6-ac3d-a5f3b13907c0" containerName="main" Oct 29 23:30:55.206850 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.206823 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.229176 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.229145 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h"] Oct 29 23:30:55.356341 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.356298 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-credential-socket\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.356564 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.356353 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-envoy\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.356564 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.356468 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istiod-ca-cert\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.356663 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.356623 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wltr\" (UniqueName: \"kubernetes.io/projected/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-kube-api-access-7wltr\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.356701 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.356678 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-workload-socket\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.356746 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.356731 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-data\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.356784 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.356761 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-podinfo\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.356821 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.356807 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-workload-certs\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.356884 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.356868 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-token\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.458582 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.458498 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wltr\" (UniqueName: \"kubernetes.io/projected/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-kube-api-access-7wltr\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.458582 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.458554 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-workload-socket\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.458790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.458588 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-data\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.458790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.458612 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-podinfo\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.458790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.458654 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-workload-certs\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.458790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.458681 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-token\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.458790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.458711 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-credential-socket\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.458790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.458765 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-envoy\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.459115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.458809 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istiod-ca-cert\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.459115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.459040 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-workload-socket\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.459115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.459088 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-data\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.459275 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.459125 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-workload-certs\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.459275 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.459170 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-credential-socket\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.459572 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.459551 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istiod-ca-cert\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.461098 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.461071 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-envoy\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.461252 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.461222 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-podinfo\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.474967 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.474944 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wltr\" (UniqueName: \"kubernetes.io/projected/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-kube-api-access-7wltr\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.475053 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.474961 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-token\") pod \"e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.521518 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.521482 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:55.839951 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.839862 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h"] Oct 29 23:30:55.845519 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:30:55.845337 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c9ffd0_41a3_47a2_84d2_5b70d9b99e5c.slice/crio-79b0b8940aa021e53ebac5b8ac3d365a9e40b67a3e4682632aa27e494a5115bc WatchSource:0}: Error finding container 79b0b8940aa021e53ebac5b8ac3d365a9e40b67a3e4682632aa27e494a5115bc: Status 404 returned error can't find the container with id 79b0b8940aa021e53ebac5b8ac3d365a9e40b67a3e4682632aa27e494a5115bc Oct 29 23:30:55.848387 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.848353 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:30:55.848498 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.848420 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:30:55.848498 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:55.848452 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:30:56.310532 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:56.310493 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" event={"ID":"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c","Type":"ContainerStarted","Data":"578a760b59cea6f1cc063bc47b2a5086b5228b24a3499e4db6363f60410c59bf"} Oct 29 23:30:56.310965 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:56.310550 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" event={"ID":"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c","Type":"ContainerStarted","Data":"79b0b8940aa021e53ebac5b8ac3d365a9e40b67a3e4682632aa27e494a5115bc"} Oct 29 23:30:56.371216 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:56.371149 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" podStartSLOduration=1.371129037 podStartE2EDuration="1.371129037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:56.367426315 +0000 UTC m=+1025.730431037" watchObservedRunningTime="2026-10-29 23:30:56.371129037 +0000 UTC m=+1025.734133763" Oct 29 23:30:56.532482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:56.524240 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:30:56.532482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:56.524386 6507 patch_prober.go:29] interesting pod/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 29 23:30:56.532482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:56.524432 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" podUID="80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 29 23:30:57.524033 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:57.523998 6507 patch_prober.go:29] interesting pod/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 29 23:30:57.524767 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:57.524726 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" podUID="80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 29 23:30:58.027565 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:58.027515 6507 patch_prober.go:29] interesting pod/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:30:58.027762 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:58.027582 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" podUID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:30:58.526484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:58.523869 6507 patch_prober.go:29] interesting pod/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 29 23:30:58.526484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:58.523932 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" podUID="80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 29 23:30:59.180024 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.179644 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:30:59.299065 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.298962 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:30:59.329680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.329647 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-data\") pod \"d02ada27-df84-4ea9-988d-3e3f0516f70a\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " Oct 29 23:30:59.330918 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.330048 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-workload-certs\") pod \"d02ada27-df84-4ea9-988d-3e3f0516f70a\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " Oct 29 23:30:59.330918 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.330153 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-workload-socket\") pod \"d02ada27-df84-4ea9-988d-3e3f0516f70a\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " Oct 29 23:30:59.331548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.331167 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d02ada27-df84-4ea9-988d-3e3f0516f70a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d02ada27-df84-4ea9-988d-3e3f0516f70a-istiod-ca-cert\") pod \"d02ada27-df84-4ea9-988d-3e3f0516f70a\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " Oct 29 23:30:59.331548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.331293 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-credential-socket\") pod \"d02ada27-df84-4ea9-988d-3e3f0516f70a\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " Oct 29 23:30:59.331548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.331369 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-envoy\") pod \"d02ada27-df84-4ea9-988d-3e3f0516f70a\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " Oct 29 23:30:59.331548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.331396 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-workload-socket" pod "d02ada27-df84-4ea9-988d-3e3f0516f70a" (UID: "d02ada27-df84-4ea9-988d-3e3f0516f70a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.331548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.331442 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02ada27-df84-4ea9-988d-3e3f0516f70a-kube-api-access-jmcxq\" (UniqueName: \"kubernetes.io/projected/d02ada27-df84-4ea9-988d-3e3f0516f70a-kube-api-access-jmcxq\") pod \"d02ada27-df84-4ea9-988d-3e3f0516f70a\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " Oct 29 23:30:59.332032 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.331640 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-data" pod "d02ada27-df84-4ea9-988d-3e3f0516f70a" (UID: "d02ada27-df84-4ea9-988d-3e3f0516f70a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.332032 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.331900 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-workload-certs" pod "d02ada27-df84-4ea9-988d-3e3f0516f70a" (UID: "d02ada27-df84-4ea9-988d-3e3f0516f70a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.332152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.332100 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-credential-socket" pod "d02ada27-df84-4ea9-988d-3e3f0516f70a" (UID: "d02ada27-df84-4ea9-988d-3e3f0516f70a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.332483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.332252 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-podinfo\") pod \"d02ada27-df84-4ea9-988d-3e3f0516f70a\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " Oct 29 23:30:59.332483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.332301 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-token\" (UniqueName: \"kubernetes.io/projected/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-token\") pod \"d02ada27-df84-4ea9-988d-3e3f0516f70a\" (UID: \"d02ada27-df84-4ea9-988d-3e3f0516f70a\") " Oct 29 23:30:59.332483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.332412 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02ada27-df84-4ea9-988d-3e3f0516f70a-istiod-ca-cert" pod "d02ada27-df84-4ea9-988d-3e3f0516f70a" (UID: "d02ada27-df84-4ea9-988d-3e3f0516f70a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:59.333140 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.333122 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.333259 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.333246 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.333357 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.333344 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d02ada27-df84-4ea9-988d-3e3f0516f70a-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.333453 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.333441 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.333865 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.333630 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.336035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.335991 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02ada27-df84-4ea9-988d-3e3f0516f70a-kube-api-access-jmcxq" pod "d02ada27-df84-4ea9-988d-3e3f0516f70a" (UID: "d02ada27-df84-4ea9-988d-3e3f0516f70a"). InnerVolumeSpecName "kube-api-access-jmcxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:59.337665 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.337451 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-podinfo" pod "d02ada27-df84-4ea9-988d-3e3f0516f70a" (UID: "d02ada27-df84-4ea9-988d-3e3f0516f70a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:30:59.338575 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.338539 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-token" pod "d02ada27-df84-4ea9-988d-3e3f0516f70a" (UID: "d02ada27-df84-4ea9-988d-3e3f0516f70a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:59.338680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.338603 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-envoy" pod "d02ada27-df84-4ea9-988d-3e3f0516f70a" (UID: "d02ada27-df84-4ea9-988d-3e3f0516f70a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.348995 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.348678 6507 generic.go:356] "Generic (PLEG): container finished" podID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" containerID="7e0c75458f20a828c3ebf7997e6f3845f0256d76897ad557ee7a9b95aa5ca6d0" exitCode=0 Oct 29 23:30:59.348995 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.348804 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" event={"ID":"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2","Type":"ContainerDied","Data":"7e0c75458f20a828c3ebf7997e6f3845f0256d76897ad557ee7a9b95aa5ca6d0"} Oct 29 23:30:59.348995 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.348872 6507 scope.go:122] "RemoveContainer" containerID="a48503f9e10bea7b5196858f404217cf7b5d7b2bb0c153df2c0030ad1748b171" Oct 29 23:30:59.358142 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.356889 6507 generic.go:356] "Generic (PLEG): container finished" podID="6451f721-f9ac-4445-b719-bd6e21820d72" containerID="efd22a131c4c954c080e1843e1f49b942f1233c71b0775d5ab24bb334534c242" exitCode=0 Oct 29 23:30:59.358142 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.357061 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" event={"ID":"6451f721-f9ac-4445-b719-bd6e21820d72","Type":"ContainerDied","Data":"efd22a131c4c954c080e1843e1f49b942f1233c71b0775d5ab24bb334534c242"} Oct 29 23:30:59.358142 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.357118 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" event={"ID":"6451f721-f9ac-4445-b719-bd6e21820d72","Type":"ContainerDied","Data":"c84df660db39ecc09571002af3cd7c1fdd1f6cdabe95b6e3fc5a437f1a4ca1de"} Oct 29 23:30:59.358142 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.357217 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf" Oct 29 23:30:59.362581 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.362551 6507 generic.go:356] "Generic (PLEG): container finished" podID="d02ada27-df84-4ea9-988d-3e3f0516f70a" containerID="c659e5a2006be61e7e90307624e50779924c7292f29de693a8c0fc0adafae3c3" exitCode=0 Oct 29 23:30:59.362944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.362916 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" event={"ID":"d02ada27-df84-4ea9-988d-3e3f0516f70a","Type":"ContainerDied","Data":"c659e5a2006be61e7e90307624e50779924c7292f29de693a8c0fc0adafae3c3"} Oct 29 23:30:59.363139 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.363111 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" event={"ID":"d02ada27-df84-4ea9-988d-3e3f0516f70a","Type":"ContainerDied","Data":"f814bdf20d3d2b3bb0309cee3fe95ee3f4f9b2abfc50f31b23fec5e2a405eb45"} Oct 29 23:30:59.363443 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.363424 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4" Oct 29 23:30:59.368924 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.368855 6507 generic.go:356] "Generic (PLEG): container finished" podID="039b4caf-d11b-4933-af15-2efb408c36e3" containerID="7633ea2be2aeb4c2d990f2afefdc49a55803208dded89996b6db4ae035bdb930" exitCode=0 Oct 29 23:30:59.368924 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.368884 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" event={"ID":"039b4caf-d11b-4933-af15-2efb408c36e3","Type":"ContainerDied","Data":"7633ea2be2aeb4c2d990f2afefdc49a55803208dded89996b6db4ae035bdb930"} Oct 29 23:30:59.379452 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.379429 6507 scope.go:122] "RemoveContainer" containerID="efd22a131c4c954c080e1843e1f49b942f1233c71b0775d5ab24bb334534c242" Oct 29 23:30:59.390221 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.390195 6507 scope.go:122] "RemoveContainer" containerID="efd22a131c4c954c080e1843e1f49b942f1233c71b0775d5ab24bb334534c242" Oct 29 23:30:59.390796 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:59.390768 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd22a131c4c954c080e1843e1f49b942f1233c71b0775d5ab24bb334534c242\": container with ID starting with efd22a131c4c954c080e1843e1f49b942f1233c71b0775d5ab24bb334534c242 not found: ID does not exist" containerID="efd22a131c4c954c080e1843e1f49b942f1233c71b0775d5ab24bb334534c242" Oct 29 23:30:59.390925 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.390808 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd22a131c4c954c080e1843e1f49b942f1233c71b0775d5ab24bb334534c242"} err="failed to get container status \"efd22a131c4c954c080e1843e1f49b942f1233c71b0775d5ab24bb334534c242\": rpc error: code = NotFound desc = could not find container \"efd22a131c4c954c080e1843e1f49b942f1233c71b0775d5ab24bb334534c242\": container with ID starting with efd22a131c4c954c080e1843e1f49b942f1233c71b0775d5ab24bb334534c242 not found: ID does not exist" Oct 29 23:30:59.390925 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.390831 6507 scope.go:122] "RemoveContainer" containerID="c659e5a2006be61e7e90307624e50779924c7292f29de693a8c0fc0adafae3c3" Oct 29 23:30:59.432553 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.432514 6507 scope.go:122] "RemoveContainer" containerID="c659e5a2006be61e7e90307624e50779924c7292f29de693a8c0fc0adafae3c3" Oct 29 23:30:59.433187 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:30:59.433104 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c659e5a2006be61e7e90307624e50779924c7292f29de693a8c0fc0adafae3c3\": container with ID starting with c659e5a2006be61e7e90307624e50779924c7292f29de693a8c0fc0adafae3c3 not found: ID does not exist" containerID="c659e5a2006be61e7e90307624e50779924c7292f29de693a8c0fc0adafae3c3" Oct 29 23:30:59.433187 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.433149 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c659e5a2006be61e7e90307624e50779924c7292f29de693a8c0fc0adafae3c3"} err="failed to get container status \"c659e5a2006be61e7e90307624e50779924c7292f29de693a8c0fc0adafae3c3\": rpc error: code = NotFound desc = could not find container \"c659e5a2006be61e7e90307624e50779924c7292f29de693a8c0fc0adafae3c3\": container with ID starting with c659e5a2006be61e7e90307624e50779924c7292f29de693a8c0fc0adafae3c3 not found: ID does not exist" Oct 29 23:30:59.434227 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.434175 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-istio-data\") pod \"6451f721-f9ac-4445-b719-bd6e21820d72\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " Oct 29 23:30:59.434356 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.434260 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-workload-socket\") pod \"6451f721-f9ac-4445-b719-bd6e21820d72\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " Oct 29 23:30:59.434356 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.434309 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6451f721-f9ac-4445-b719-bd6e21820d72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6451f721-f9ac-4445-b719-bd6e21820d72-istiod-ca-cert\") pod \"6451f721-f9ac-4445-b719-bd6e21820d72\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " Oct 29 23:30:59.434494 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.434368 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-istio-envoy\") pod \"6451f721-f9ac-4445-b719-bd6e21820d72\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " Oct 29 23:30:59.434494 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.434400 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-istio-data" pod "6451f721-f9ac-4445-b719-bd6e21820d72" (UID: "6451f721-f9ac-4445-b719-bd6e21820d72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.434494 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.434415 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6451f721-f9ac-4445-b719-bd6e21820d72-kube-api-access-fvgxk\" (UniqueName: \"kubernetes.io/projected/6451f721-f9ac-4445-b719-bd6e21820d72-kube-api-access-fvgxk\") pod \"6451f721-f9ac-4445-b719-bd6e21820d72\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " Oct 29 23:30:59.434494 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.434471 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-credential-socket\") pod \"6451f721-f9ac-4445-b719-bd6e21820d72\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " Oct 29 23:30:59.434710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.434545 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6451f721-f9ac-4445-b719-bd6e21820d72-istio-token\" (UniqueName: \"kubernetes.io/projected/6451f721-f9ac-4445-b719-bd6e21820d72-istio-token\") pod \"6451f721-f9ac-4445-b719-bd6e21820d72\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " Oct 29 23:30:59.434710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.434613 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6451f721-f9ac-4445-b719-bd6e21820d72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6451f721-f9ac-4445-b719-bd6e21820d72-istio-podinfo\") pod \"6451f721-f9ac-4445-b719-bd6e21820d72\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " Oct 29 23:30:59.434710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.434686 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-workload-certs\") pod \"6451f721-f9ac-4445-b719-bd6e21820d72\" (UID: \"6451f721-f9ac-4445-b719-bd6e21820d72\") " Oct 29 23:30:59.434887 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.434866 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6451f721-f9ac-4445-b719-bd6e21820d72-istiod-ca-cert" pod "6451f721-f9ac-4445-b719-bd6e21820d72" (UID: "6451f721-f9ac-4445-b719-bd6e21820d72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:59.435174 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.435128 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-workload-certs" pod "6451f721-f9ac-4445-b719-bd6e21820d72" (UID: "6451f721-f9ac-4445-b719-bd6e21820d72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.435578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.435470 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.435578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.435498 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.435578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.435513 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.435578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.435526 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmcxq\" (UniqueName: \"kubernetes.io/projected/d02ada27-df84-4ea9-988d-3e3f0516f70a-kube-api-access-jmcxq\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.435578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.435544 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.435578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.435558 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d02ada27-df84-4ea9-988d-3e3f0516f70a-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.435578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.435570 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6451f721-f9ac-4445-b719-bd6e21820d72-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.435981 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.435597 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4"] Oct 29 23:30:59.435981 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.435653 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-workload-socket" pod "6451f721-f9ac-4445-b719-bd6e21820d72" (UID: "6451f721-f9ac-4445-b719-bd6e21820d72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.435981 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.435756 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-credential-socket" pod "6451f721-f9ac-4445-b719-bd6e21820d72" (UID: "6451f721-f9ac-4445-b719-bd6e21820d72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.437291 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.437262 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:30:59.438207 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.438170 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-istio-envoy" pod "6451f721-f9ac-4445-b719-bd6e21820d72" (UID: "6451f721-f9ac-4445-b719-bd6e21820d72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.438518 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.438488 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6451f721-f9ac-4445-b719-bd6e21820d72-kube-api-access-fvgxk" pod "6451f721-f9ac-4445-b719-bd6e21820d72" (UID: "6451f721-f9ac-4445-b719-bd6e21820d72"). InnerVolumeSpecName "kube-api-access-fvgxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:59.440413 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.440374 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-b2jg4"] Oct 29 23:30:59.440933 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.440765 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6451f721-f9ac-4445-b719-bd6e21820d72-istio-token" pod "6451f721-f9ac-4445-b719-bd6e21820d72" (UID: "6451f721-f9ac-4445-b719-bd6e21820d72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:59.441600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.441577 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6451f721-f9ac-4445-b719-bd6e21820d72-istio-podinfo" pod "6451f721-f9ac-4445-b719-bd6e21820d72" (UID: "6451f721-f9ac-4445-b719-bd6e21820d72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:30:59.443363 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.443307 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:30:59.537593 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.537038 6507 patch_prober.go:29] interesting pod/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 29 23:30:59.537593 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.537086 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" podUID="80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 29 23:30:59.537593 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.537427 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/039b4caf-d11b-4933-af15-2efb408c36e3-istio-token\" (UniqueName: \"kubernetes.io/projected/039b4caf-d11b-4933-af15-2efb408c36e3-istio-token\") pod \"039b4caf-d11b-4933-af15-2efb408c36e3\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " Oct 29 23:30:59.537593 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.537532 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-envoy\") pod \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " Oct 29 23:30:59.538234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.537606 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-token\" (UniqueName: \"kubernetes.io/projected/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-token\") pod \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " Oct 29 23:30:59.538234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.537637 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-workload-socket\") pod \"039b4caf-d11b-4933-af15-2efb408c36e3\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " Oct 29 23:30:59.538234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.537672 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-workload-socket\") pod \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " Oct 29 23:30:59.538234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.537723 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-data\") pod \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " Oct 29 23:30:59.538234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.537755 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-istio-envoy\") pod \"039b4caf-d11b-4933-af15-2efb408c36e3\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " Oct 29 23:30:59.538234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.537783 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-istio-data\") pod \"039b4caf-d11b-4933-af15-2efb408c36e3\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " Oct 29 23:30:59.538234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.537819 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-podinfo\") pod \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " Oct 29 23:30:59.538234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.537876 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/039b4caf-d11b-4933-af15-2efb408c36e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/039b4caf-d11b-4933-af15-2efb408c36e3-istiod-ca-cert\") pod \"039b4caf-d11b-4933-af15-2efb408c36e3\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " Oct 29 23:30:59.538234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.537909 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/039b4caf-d11b-4933-af15-2efb408c36e3-kube-api-access-v8b29\" (UniqueName: \"kubernetes.io/projected/039b4caf-d11b-4933-af15-2efb408c36e3-kube-api-access-v8b29\") pod \"039b4caf-d11b-4933-af15-2efb408c36e3\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " Oct 29 23:30:59.538234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.537978 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-workload-certs\") pod \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " Oct 29 23:30:59.538234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.538044 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-kube-api-access-8z7gt\" (UniqueName: \"kubernetes.io/projected/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-kube-api-access-8z7gt\") pod \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " Oct 29 23:30:59.538234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.538106 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/039b4caf-d11b-4933-af15-2efb408c36e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/039b4caf-d11b-4933-af15-2efb408c36e3-istio-podinfo\") pod \"039b4caf-d11b-4933-af15-2efb408c36e3\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " Oct 29 23:30:59.538234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.538146 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-credential-socket\") pod \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " Oct 29 23:30:59.538234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.538200 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-workload-certs\") pod \"039b4caf-d11b-4933-af15-2efb408c36e3\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " Oct 29 23:30:59.538234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.538241 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istiod-ca-cert\") pod \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\" (UID: \"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2\") " Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.538286 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-credential-socket\") pod \"039b4caf-d11b-4933-af15-2efb408c36e3\" (UID: \"039b4caf-d11b-4933-af15-2efb408c36e3\") " Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.538750 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-workload-socket" pod "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" (UID: "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.538885 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.538907 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6451f721-f9ac-4445-b719-bd6e21820d72-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.538924 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.539010 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-credential-socket" pod "039b4caf-d11b-4933-af15-2efb408c36e3" (UID: "039b4caf-d11b-4933-af15-2efb408c36e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.539328 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/039b4caf-d11b-4933-af15-2efb408c36e3-istiod-ca-cert" pod "039b4caf-d11b-4933-af15-2efb408c36e3" (UID: "039b4caf-d11b-4933-af15-2efb408c36e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.539329 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-workload-socket" pod "039b4caf-d11b-4933-af15-2efb408c36e3" (UID: "039b4caf-d11b-4933-af15-2efb408c36e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.540056 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039b4caf-d11b-4933-af15-2efb408c36e3-istio-token" pod "039b4caf-d11b-4933-af15-2efb408c36e3" (UID: "039b4caf-d11b-4933-af15-2efb408c36e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.540249 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-workload-certs" pod "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" (UID: "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.540291 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-credential-socket" pod "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" (UID: "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.538941 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.541619 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvgxk\" (UniqueName: \"kubernetes.io/projected/6451f721-f9ac-4445-b719-bd6e21820d72-kube-api-access-fvgxk\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.541638 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6451f721-f9ac-4445-b719-bd6e21820d72-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.541655 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6451f721-f9ac-4445-b719-bd6e21820d72-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.541839 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-data" pod "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" (UID: "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.541866 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-workload-certs" pod "039b4caf-d11b-4933-af15-2efb408c36e3" (UID: "039b4caf-d11b-4933-af15-2efb408c36e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.541929 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-podinfo" pod "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" (UID: "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.542164 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istiod-ca-cert" pod "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" (UID: "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:59.542775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.542640 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-istio-data" pod "039b4caf-d11b-4933-af15-2efb408c36e3" (UID: "039b4caf-d11b-4933-af15-2efb408c36e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.543999 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.543976 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-envoy" pod "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" (UID: "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.544055 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.544030 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/039b4caf-d11b-4933-af15-2efb408c36e3-istio-podinfo" pod "039b4caf-d11b-4933-af15-2efb408c36e3" (UID: "039b4caf-d11b-4933-af15-2efb408c36e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:30:59.544162 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.544141 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039b4caf-d11b-4933-af15-2efb408c36e3-kube-api-access-v8b29" pod "039b4caf-d11b-4933-af15-2efb408c36e3" (UID: "039b4caf-d11b-4933-af15-2efb408c36e3"). InnerVolumeSpecName "kube-api-access-v8b29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:59.544962 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.544928 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-kube-api-access-8z7gt" pod "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" (UID: "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2"). InnerVolumeSpecName "kube-api-access-8z7gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:59.545079 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.545006 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-istio-envoy" pod "039b4caf-d11b-4933-af15-2efb408c36e3" (UID: "039b4caf-d11b-4933-af15-2efb408c36e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:30:59.545139 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.545079 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-token" pod "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" (UID: "dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:59.643349 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643313 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z7gt\" (UniqueName: \"kubernetes.io/projected/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-kube-api-access-8z7gt\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.643349 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643349 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/039b4caf-d11b-4933-af15-2efb408c36e3-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.643602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643359 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.643602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643368 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.643602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643377 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.643602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643385 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.643602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643393 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/039b4caf-d11b-4933-af15-2efb408c36e3-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.643602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643401 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.643602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643408 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.643602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643415 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.643602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643423 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.643602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643431 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.643602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643438 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/039b4caf-d11b-4933-af15-2efb408c36e3-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.643602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643445 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.643602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643470 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/039b4caf-d11b-4933-af15-2efb408c36e3-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.643602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643483 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8b29\" (UniqueName: \"kubernetes.io/projected/039b4caf-d11b-4933-af15-2efb408c36e3-kube-api-access-v8b29\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.643602 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.643496 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:30:59.850495 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.845863 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf"] Oct 29 23:30:59.863952 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.863920 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-7rwgf"] Oct 29 23:30:59.871700 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:30:59.871661 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:31:00.001106 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.001079 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:31:00.046777 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.046693 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:31:00.049900 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.049874 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-workload-certs\") pod \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " Oct 29 23:31:00.050058 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.049974 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-token\" (UniqueName: \"kubernetes.io/projected/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-token\") pod \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " Oct 29 23:31:00.050058 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.050015 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-envoy\") pod \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " Oct 29 23:31:00.050170 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.050084 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-podinfo\") pod \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " Oct 29 23:31:00.050170 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.050116 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-kube-api-access-fpm4g\" (UniqueName: \"kubernetes.io/projected/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-kube-api-access-fpm4g\") pod \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " Oct 29 23:31:00.050170 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.050139 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-credential-socket\") pod \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " Oct 29 23:31:00.050330 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.050201 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-data\") pod \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " Oct 29 23:31:00.050330 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.050232 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istiod-ca-cert\") pod \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " Oct 29 23:31:00.050330 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.050256 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-workload-socket\") pod \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\" (UID: \"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4\") " Oct 29 23:31:00.050514 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.050334 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-workload-certs" pod "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" (UID: "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:00.050585 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.050543 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-credential-socket" pod "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" (UID: "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:00.050731 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.050707 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-workload-socket" pod "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" (UID: "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:00.050800 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.050716 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-data" pod "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" (UID: "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:00.050800 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.050789 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.050903 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.050813 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.051009 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.050989 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istiod-ca-cert" pod "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" (UID: "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:00.052685 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.052663 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-token" pod "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" (UID: "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:00.052746 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.052691 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-podinfo" pod "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" (UID: "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:31:00.052782 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.052746 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-kube-api-access-fpm4g" pod "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" (UID: "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4"). InnerVolumeSpecName "kube-api-access-fpm4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:00.052782 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.052754 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-envoy" pod "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" (UID: "2c28cad9-e9e2-4e39-bc47-40047ca0a7e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:00.152029 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.151996 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-credential-socket\") pod \"12f4ed99-042f-4e41-8489-9af7622e632c\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " Oct 29 23:31:00.152173 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152087 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-istio-data\") pod \"12f4ed99-042f-4e41-8489-9af7622e632c\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " Oct 29 23:31:00.152173 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152122 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-istio-envoy\") pod \"aa280316-fb6c-40db-8658-65b84bb4382b\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " Oct 29 23:31:00.152261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152173 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa280316-fb6c-40db-8658-65b84bb4382b-kube-api-access-pc4d9\" (UniqueName: \"kubernetes.io/projected/aa280316-fb6c-40db-8658-65b84bb4382b-kube-api-access-pc4d9\") pod \"aa280316-fb6c-40db-8658-65b84bb4382b\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " Oct 29 23:31:00.152261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152205 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12f4ed99-042f-4e41-8489-9af7622e632c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f4ed99-042f-4e41-8489-9af7622e632c-istiod-ca-cert\") pod \"12f4ed99-042f-4e41-8489-9af7622e632c\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " Oct 29 23:31:00.152261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152247 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-workload-socket\") pod \"12f4ed99-042f-4e41-8489-9af7622e632c\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " Oct 29 23:31:00.152373 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152285 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-workload-socket\") pod \"aa280316-fb6c-40db-8658-65b84bb4382b\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " Oct 29 23:31:00.152373 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152312 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa280316-fb6c-40db-8658-65b84bb4382b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa280316-fb6c-40db-8658-65b84bb4382b-istiod-ca-cert\") pod \"aa280316-fb6c-40db-8658-65b84bb4382b\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " Oct 29 23:31:00.152373 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152341 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-credential-socket\") pod \"aa280316-fb6c-40db-8658-65b84bb4382b\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " Oct 29 23:31:00.152373 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152339 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-credential-socket" pod "12f4ed99-042f-4e41-8489-9af7622e632c" (UID: "12f4ed99-042f-4e41-8489-9af7622e632c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:00.152539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152372 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-workload-certs\") pod \"aa280316-fb6c-40db-8658-65b84bb4382b\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " Oct 29 23:31:00.152539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152401 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-istio-envoy\") pod \"12f4ed99-042f-4e41-8489-9af7622e632c\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " Oct 29 23:31:00.152539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152431 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f4ed99-042f-4e41-8489-9af7622e632c-kube-api-access-h2qdv\" (UniqueName: \"kubernetes.io/projected/12f4ed99-042f-4e41-8489-9af7622e632c-kube-api-access-h2qdv\") pod \"12f4ed99-042f-4e41-8489-9af7622e632c\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " Oct 29 23:31:00.152539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152521 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa280316-fb6c-40db-8658-65b84bb4382b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa280316-fb6c-40db-8658-65b84bb4382b-istio-podinfo\") pod \"aa280316-fb6c-40db-8658-65b84bb4382b\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " Oct 29 23:31:00.152754 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152559 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-istio-data\") pod \"aa280316-fb6c-40db-8658-65b84bb4382b\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " Oct 29 23:31:00.152754 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152668 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa280316-fb6c-40db-8658-65b84bb4382b-istio-token\" (UniqueName: \"kubernetes.io/projected/aa280316-fb6c-40db-8658-65b84bb4382b-istio-token\") pod \"aa280316-fb6c-40db-8658-65b84bb4382b\" (UID: \"aa280316-fb6c-40db-8658-65b84bb4382b\") " Oct 29 23:31:00.152754 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152719 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12f4ed99-042f-4e41-8489-9af7622e632c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f4ed99-042f-4e41-8489-9af7622e632c-istio-podinfo\") pod \"12f4ed99-042f-4e41-8489-9af7622e632c\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " Oct 29 23:31:00.152754 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152748 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-workload-certs\") pod \"12f4ed99-042f-4e41-8489-9af7622e632c\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " Oct 29 23:31:00.152947 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152758 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-workload-certs" pod "aa280316-fb6c-40db-8658-65b84bb4382b" (UID: "aa280316-fb6c-40db-8658-65b84bb4382b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:00.152947 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152776 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-credential-socket" pod "aa280316-fb6c-40db-8658-65b84bb4382b" (UID: "aa280316-fb6c-40db-8658-65b84bb4382b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:00.152947 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152781 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-workload-socket" pod "12f4ed99-042f-4e41-8489-9af7622e632c" (UID: "12f4ed99-042f-4e41-8489-9af7622e632c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:00.152947 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.152787 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f4ed99-042f-4e41-8489-9af7622e632c-istio-token\" (UniqueName: \"kubernetes.io/projected/12f4ed99-042f-4e41-8489-9af7622e632c-istio-token\") pod \"12f4ed99-042f-4e41-8489-9af7622e632c\" (UID: \"12f4ed99-042f-4e41-8489-9af7622e632c\") " Oct 29 23:31:00.153145 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153005 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-istio-data" pod "aa280316-fb6c-40db-8658-65b84bb4382b" (UID: "aa280316-fb6c-40db-8658-65b84bb4382b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:00.153145 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153103 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f4ed99-042f-4e41-8489-9af7622e632c-istiod-ca-cert" pod "12f4ed99-042f-4e41-8489-9af7622e632c" (UID: "12f4ed99-042f-4e41-8489-9af7622e632c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:00.153702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153411 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-istio-data" pod "12f4ed99-042f-4e41-8489-9af7622e632c" (UID: "12f4ed99-042f-4e41-8489-9af7622e632c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:00.153702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153448 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-workload-socket" pod "aa280316-fb6c-40db-8658-65b84bb4382b" (UID: "aa280316-fb6c-40db-8658-65b84bb4382b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:00.153702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153506 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.153702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153529 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.153702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153543 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.153702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153558 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.153702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153573 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.153702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153588 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpm4g\" (UniqueName: \"kubernetes.io/projected/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-kube-api-access-fpm4g\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.153702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153604 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.153702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153618 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.153702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153630 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.153702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153643 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f4ed99-042f-4e41-8489-9af7622e632c-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.153702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153658 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.153702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153671 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.153702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153684 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.154680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.153892 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-workload-certs" pod "12f4ed99-042f-4e41-8489-9af7622e632c" (UID: "12f4ed99-042f-4e41-8489-9af7622e632c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:00.155173 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.155144 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa280316-fb6c-40db-8658-65b84bb4382b-kube-api-access-pc4d9" pod "aa280316-fb6c-40db-8658-65b84bb4382b" (UID: "aa280316-fb6c-40db-8658-65b84bb4382b"). InnerVolumeSpecName "kube-api-access-pc4d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:00.155372 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.155347 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f4ed99-042f-4e41-8489-9af7622e632c-istio-token" pod "12f4ed99-042f-4e41-8489-9af7622e632c" (UID: "12f4ed99-042f-4e41-8489-9af7622e632c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:00.155573 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.155549 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa280316-fb6c-40db-8658-65b84bb4382b-istiod-ca-cert" pod "aa280316-fb6c-40db-8658-65b84bb4382b" (UID: "aa280316-fb6c-40db-8658-65b84bb4382b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:00.155710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.155686 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12f4ed99-042f-4e41-8489-9af7622e632c-istio-podinfo" pod "12f4ed99-042f-4e41-8489-9af7622e632c" (UID: "12f4ed99-042f-4e41-8489-9af7622e632c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:31:00.155816 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.155797 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa280316-fb6c-40db-8658-65b84bb4382b-istio-podinfo" pod "aa280316-fb6c-40db-8658-65b84bb4382b" (UID: "aa280316-fb6c-40db-8658-65b84bb4382b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:31:00.156213 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.156193 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa280316-fb6c-40db-8658-65b84bb4382b-istio-token" pod "aa280316-fb6c-40db-8658-65b84bb4382b" (UID: "aa280316-fb6c-40db-8658-65b84bb4382b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:00.156740 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.156717 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-istio-envoy" pod "12f4ed99-042f-4e41-8489-9af7622e632c" (UID: "12f4ed99-042f-4e41-8489-9af7622e632c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:00.157831 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.157801 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f4ed99-042f-4e41-8489-9af7622e632c-kube-api-access-h2qdv" pod "12f4ed99-042f-4e41-8489-9af7622e632c" (UID: "12f4ed99-042f-4e41-8489-9af7622e632c"). InnerVolumeSpecName "kube-api-access-h2qdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:00.158179 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.158150 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-istio-envoy" pod "aa280316-fb6c-40db-8658-65b84bb4382b" (UID: "aa280316-fb6c-40db-8658-65b84bb4382b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:00.254871 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.254839 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa280316-fb6c-40db-8658-65b84bb4382b-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.254871 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.254868 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f4ed99-042f-4e41-8489-9af7622e632c-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.254871 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.254877 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.255097 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.254885 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f4ed99-042f-4e41-8489-9af7622e632c-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.255097 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.254896 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.255097 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.254903 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.255097 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.254912 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc4d9\" (UniqueName: \"kubernetes.io/projected/aa280316-fb6c-40db-8658-65b84bb4382b-kube-api-access-pc4d9\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.255097 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.254920 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa280316-fb6c-40db-8658-65b84bb4382b-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.255097 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.254930 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa280316-fb6c-40db-8658-65b84bb4382b-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.255097 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.254940 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f4ed99-042f-4e41-8489-9af7622e632c-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.255097 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.254947 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2qdv\" (UniqueName: \"kubernetes.io/projected/12f4ed99-042f-4e41-8489-9af7622e632c-kube-api-access-h2qdv\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.255097 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.254955 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa280316-fb6c-40db-8658-65b84bb4382b-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:00.378184 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.378100 6507 generic.go:356] "Generic (PLEG): container finished" podID="12f4ed99-042f-4e41-8489-9af7622e632c" containerID="2a6cecff018d66364ad7f5d48d752ab3eb2f23b50f5814c81aa66387bfa2ee53" exitCode=0 Oct 29 23:31:00.378184 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.378148 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" Oct 29 23:31:00.378417 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.378184 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" event={"ID":"12f4ed99-042f-4e41-8489-9af7622e632c","Type":"ContainerDied","Data":"2a6cecff018d66364ad7f5d48d752ab3eb2f23b50f5814c81aa66387bfa2ee53"} Oct 29 23:31:00.378417 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.378252 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2" event={"ID":"12f4ed99-042f-4e41-8489-9af7622e632c","Type":"ContainerDied","Data":"b4641f1981bddb9c4153fcbdb927234c8e3c85d8b1f42b630f85a415264103ad"} Oct 29 23:31:00.378417 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.378283 6507 scope.go:122] "RemoveContainer" containerID="2a6cecff018d66364ad7f5d48d752ab3eb2f23b50f5814c81aa66387bfa2ee53" Oct 29 23:31:00.379945 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.379920 6507 generic.go:356] "Generic (PLEG): container finished" podID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" containerID="af8b4d7dcd6f28d79776ff315ddc2e0678c21bb4d91d4cd2d4f6fcd34ab88227" exitCode=0 Oct 29 23:31:00.380049 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.379948 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" event={"ID":"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4","Type":"ContainerDied","Data":"af8b4d7dcd6f28d79776ff315ddc2e0678c21bb4d91d4cd2d4f6fcd34ab88227"} Oct 29 23:31:00.380049 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.379963 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" Oct 29 23:31:00.380049 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.380005 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5" event={"ID":"2c28cad9-e9e2-4e39-bc47-40047ca0a7e4","Type":"ContainerDied","Data":"8f55cc699504f6839fc6d42efbc30e0aebcad019487ad006d8e2a21e2ef1e080"} Oct 29 23:31:00.382605 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.382575 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" Oct 29 23:31:00.382699 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.382598 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6" event={"ID":"039b4caf-d11b-4933-af15-2efb408c36e3","Type":"ContainerDied","Data":"6c93b60d4cf073aaa13cd4129ea17273a80dcd3b46b9fed258ae8be21831b58f"} Oct 29 23:31:00.385346 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.384840 6507 generic.go:356] "Generic (PLEG): container finished" podID="aa280316-fb6c-40db-8658-65b84bb4382b" containerID="d6a25edbc79d32ff0fdc5947f6f45dfd61e520e49b93b29de043754e00bab8bd" exitCode=0 Oct 29 23:31:00.385346 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.384887 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" Oct 29 23:31:00.385346 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.384964 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" event={"ID":"aa280316-fb6c-40db-8658-65b84bb4382b","Type":"ContainerDied","Data":"d6a25edbc79d32ff0fdc5947f6f45dfd61e520e49b93b29de043754e00bab8bd"} Oct 29 23:31:00.385346 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.385009 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4" event={"ID":"aa280316-fb6c-40db-8658-65b84bb4382b","Type":"ContainerDied","Data":"5f098b073283e5725390db5b3a94517fcae982acad7e40a034b50ac43afb445b"} Oct 29 23:31:00.392488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.389966 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" Oct 29 23:31:00.392488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.390170 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr" event={"ID":"dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2","Type":"ContainerDied","Data":"a41e6cc35cb00e23a9d35dc08d0f3458b975de49ee4e58111ee5e7e44ebcf8ce"} Oct 29 23:31:00.392488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.391213 6507 scope.go:122] "RemoveContainer" containerID="2a6cecff018d66364ad7f5d48d752ab3eb2f23b50f5814c81aa66387bfa2ee53" Oct 29 23:31:00.392488 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:31:00.391908 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6cecff018d66364ad7f5d48d752ab3eb2f23b50f5814c81aa66387bfa2ee53\": container with ID starting with 2a6cecff018d66364ad7f5d48d752ab3eb2f23b50f5814c81aa66387bfa2ee53 not found: ID does not exist" containerID="2a6cecff018d66364ad7f5d48d752ab3eb2f23b50f5814c81aa66387bfa2ee53" Oct 29 23:31:00.392488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.391941 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6cecff018d66364ad7f5d48d752ab3eb2f23b50f5814c81aa66387bfa2ee53"} err="failed to get container status \"2a6cecff018d66364ad7f5d48d752ab3eb2f23b50f5814c81aa66387bfa2ee53\": rpc error: code = NotFound desc = could not find container \"2a6cecff018d66364ad7f5d48d752ab3eb2f23b50f5814c81aa66387bfa2ee53\": container with ID starting with 2a6cecff018d66364ad7f5d48d752ab3eb2f23b50f5814c81aa66387bfa2ee53 not found: ID does not exist" Oct 29 23:31:00.392488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.391964 6507 scope.go:122] "RemoveContainer" containerID="af8b4d7dcd6f28d79776ff315ddc2e0678c21bb4d91d4cd2d4f6fcd34ab88227" Oct 29 23:31:00.431766 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.431223 6507 scope.go:122] "RemoveContainer" containerID="af8b4d7dcd6f28d79776ff315ddc2e0678c21bb4d91d4cd2d4f6fcd34ab88227" Oct 29 23:31:00.432101 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:31:00.432070 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af8b4d7dcd6f28d79776ff315ddc2e0678c21bb4d91d4cd2d4f6fcd34ab88227\": container with ID starting with af8b4d7dcd6f28d79776ff315ddc2e0678c21bb4d91d4cd2d4f6fcd34ab88227 not found: ID does not exist" containerID="af8b4d7dcd6f28d79776ff315ddc2e0678c21bb4d91d4cd2d4f6fcd34ab88227" Oct 29 23:31:00.432219 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.432198 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af8b4d7dcd6f28d79776ff315ddc2e0678c21bb4d91d4cd2d4f6fcd34ab88227"} err="failed to get container status \"af8b4d7dcd6f28d79776ff315ddc2e0678c21bb4d91d4cd2d4f6fcd34ab88227\": rpc error: code = NotFound desc = could not find container \"af8b4d7dcd6f28d79776ff315ddc2e0678c21bb4d91d4cd2d4f6fcd34ab88227\": container with ID starting with af8b4d7dcd6f28d79776ff315ddc2e0678c21bb4d91d4cd2d4f6fcd34ab88227 not found: ID does not exist" Oct 29 23:31:00.432310 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.432292 6507 scope.go:122] "RemoveContainer" containerID="7633ea2be2aeb4c2d990f2afefdc49a55803208dded89996b6db4ae035bdb930" Oct 29 23:31:00.444784 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.444759 6507 scope.go:122] "RemoveContainer" containerID="d6a25edbc79d32ff0fdc5947f6f45dfd61e520e49b93b29de043754e00bab8bd" Oct 29 23:31:00.455227 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.455189 6507 scope.go:122] "RemoveContainer" containerID="369f12f078ada2771f8e0a12bb8823e4ef31052e6dede14826fd54975da29523" Oct 29 23:31:00.544756 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.544722 6507 patch_prober.go:29] interesting pod/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 29 23:31:00.545287 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.545258 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" podUID="80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 29 23:31:00.557711 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.557601 6507 scope.go:122] "RemoveContainer" containerID="d6a25edbc79d32ff0fdc5947f6f45dfd61e520e49b93b29de043754e00bab8bd" Oct 29 23:31:00.558201 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:31:00.558161 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a25edbc79d32ff0fdc5947f6f45dfd61e520e49b93b29de043754e00bab8bd\": container with ID starting with d6a25edbc79d32ff0fdc5947f6f45dfd61e520e49b93b29de043754e00bab8bd not found: ID does not exist" containerID="d6a25edbc79d32ff0fdc5947f6f45dfd61e520e49b93b29de043754e00bab8bd" Oct 29 23:31:00.558397 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.558320 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a25edbc79d32ff0fdc5947f6f45dfd61e520e49b93b29de043754e00bab8bd"} err="failed to get container status \"d6a25edbc79d32ff0fdc5947f6f45dfd61e520e49b93b29de043754e00bab8bd\": rpc error: code = NotFound desc = could not find container \"d6a25edbc79d32ff0fdc5947f6f45dfd61e520e49b93b29de043754e00bab8bd\": container with ID starting with d6a25edbc79d32ff0fdc5947f6f45dfd61e520e49b93b29de043754e00bab8bd not found: ID does not exist" Oct 29 23:31:00.558397 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.558344 6507 scope.go:122] "RemoveContainer" containerID="369f12f078ada2771f8e0a12bb8823e4ef31052e6dede14826fd54975da29523" Oct 29 23:31:00.558807 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:31:00.558783 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"369f12f078ada2771f8e0a12bb8823e4ef31052e6dede14826fd54975da29523\": container with ID starting with 369f12f078ada2771f8e0a12bb8823e4ef31052e6dede14826fd54975da29523 not found: ID does not exist" containerID="369f12f078ada2771f8e0a12bb8823e4ef31052e6dede14826fd54975da29523" Oct 29 23:31:00.558940 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.558914 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"369f12f078ada2771f8e0a12bb8823e4ef31052e6dede14826fd54975da29523"} err="failed to get container status \"369f12f078ada2771f8e0a12bb8823e4ef31052e6dede14826fd54975da29523\": rpc error: code = NotFound desc = could not find container \"369f12f078ada2771f8e0a12bb8823e4ef31052e6dede14826fd54975da29523\": container with ID starting with 369f12f078ada2771f8e0a12bb8823e4ef31052e6dede14826fd54975da29523 not found: ID does not exist" Oct 29 23:31:00.559028 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.559019 6507 scope.go:122] "RemoveContainer" containerID="7e0c75458f20a828c3ebf7997e6f3845f0256d76897ad557ee7a9b95aa5ca6d0" Oct 29 23:31:00.575150 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.575120 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr"] Oct 29 23:31:00.585314 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.585285 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-wmcnr"] Oct 29 23:31:00.599929 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.599878 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5"] Oct 29 23:31:00.624494 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.621221 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-ldgz5"] Oct 29 23:31:00.641867 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.641783 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4"] Oct 29 23:31:00.648618 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.648582 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-6fwb4"] Oct 29 23:31:00.677045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.677010 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2"] Oct 29 23:31:00.683578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.683550 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-837245-openshift-default-67857c7775-zb8p2"] Oct 29 23:31:00.728930 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.728888 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6"] Oct 29 23:31:00.734172 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.734136 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2670a5-openshift-default-9f9b57bbc-lndh6"] Oct 29 23:31:00.754090 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.754056 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="039b4caf-d11b-4933-af15-2efb408c36e3" path="/var/lib/kubelet/pods/039b4caf-d11b-4933-af15-2efb408c36e3/volumes" Oct 29 23:31:00.754576 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.754560 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12f4ed99-042f-4e41-8489-9af7622e632c" path="/var/lib/kubelet/pods/12f4ed99-042f-4e41-8489-9af7622e632c/volumes" Oct 29 23:31:00.755020 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.755007 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" path="/var/lib/kubelet/pods/2c28cad9-e9e2-4e39-bc47-40047ca0a7e4/volumes" Oct 29 23:31:00.755766 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.755752 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" path="/var/lib/kubelet/pods/6451f721-f9ac-4445-b719-bd6e21820d72/volumes" Oct 29 23:31:00.756170 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.756158 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa280316-fb6c-40db-8658-65b84bb4382b" path="/var/lib/kubelet/pods/aa280316-fb6c-40db-8658-65b84bb4382b/volumes" Oct 29 23:31:00.756625 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.756611 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d02ada27-df84-4ea9-988d-3e3f0516f70a" path="/var/lib/kubelet/pods/d02ada27-df84-4ea9-988d-3e3f0516f70a/volumes" Oct 29 23:31:00.757295 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.757282 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" path="/var/lib/kubelet/pods/dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2/volumes" Oct 29 23:31:00.930719 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.930621 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr"] Oct 29 23:31:00.931366 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931345 6507 cpu_manager.go:414] "containerMap: removing container" podUID="aa280316-fb6c-40db-8658-65b84bb4382b" containerName="istio-proxy" Oct 29 23:31:00.931409 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931371 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa280316-fb6c-40db-8658-65b84bb4382b" containerName="istio-proxy" Oct 29 23:31:00.931409 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931396 6507 cpu_manager.go:414] "containerMap: removing container" podUID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" containerName="istio-proxy" Oct 29 23:31:00.931409 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931404 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" containerName="istio-proxy" Oct 29 23:31:00.931535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931420 6507 cpu_manager.go:414] "containerMap: removing container" podUID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" containerName="istio-proxy" Oct 29 23:31:00.931535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931429 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" containerName="istio-proxy" Oct 29 23:31:00.931535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931473 6507 cpu_manager.go:414] "containerMap: removing container" podUID="039b4caf-d11b-4933-af15-2efb408c36e3" containerName="istio-proxy" Oct 29 23:31:00.931535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931481 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="039b4caf-d11b-4933-af15-2efb408c36e3" containerName="istio-proxy" Oct 29 23:31:00.931535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931501 6507 cpu_manager.go:414] "containerMap: removing container" podUID="aa280316-fb6c-40db-8658-65b84bb4382b" containerName="istio-proxy" Oct 29 23:31:00.931535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931506 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa280316-fb6c-40db-8658-65b84bb4382b" containerName="istio-proxy" Oct 29 23:31:00.931535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931514 6507 cpu_manager.go:414] "containerMap: removing container" podUID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" containerName="istio-proxy" Oct 29 23:31:00.931535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931519 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" containerName="istio-proxy" Oct 29 23:31:00.931535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931526 6507 cpu_manager.go:414] "containerMap: removing container" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" containerName="istio-proxy" Oct 29 23:31:00.931535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931529 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" containerName="istio-proxy" Oct 29 23:31:00.931535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931537 6507 cpu_manager.go:414] "containerMap: removing container" podUID="d02ada27-df84-4ea9-988d-3e3f0516f70a" containerName="istio-proxy" Oct 29 23:31:00.931847 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931541 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d02ada27-df84-4ea9-988d-3e3f0516f70a" containerName="istio-proxy" Oct 29 23:31:00.931847 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931548 6507 cpu_manager.go:414] "containerMap: removing container" podUID="12f4ed99-042f-4e41-8489-9af7622e632c" containerName="istio-proxy" Oct 29 23:31:00.931847 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931552 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12f4ed99-042f-4e41-8489-9af7622e632c" containerName="istio-proxy" Oct 29 23:31:00.931847 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931634 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa280316-fb6c-40db-8658-65b84bb4382b" containerName="istio-proxy" Oct 29 23:31:00.931847 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931642 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" containerName="istio-proxy" Oct 29 23:31:00.931847 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931649 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbdb5a2-f2e7-4d1b-946a-8d35cacdb9e2" containerName="istio-proxy" Oct 29 23:31:00.931847 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931658 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="039b4caf-d11b-4933-af15-2efb408c36e3" containerName="istio-proxy" Oct 29 23:31:00.931847 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931666 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c28cad9-e9e2-4e39-bc47-40047ca0a7e4" containerName="istio-proxy" Oct 29 23:31:00.931847 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931672 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="6451f721-f9ac-4445-b719-bd6e21820d72" containerName="istio-proxy" Oct 29 23:31:00.931847 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931679 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="d02ada27-df84-4ea9-988d-3e3f0516f70a" containerName="istio-proxy" Oct 29 23:31:00.931847 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.931684 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="12f4ed99-042f-4e41-8489-9af7622e632c" containerName="istio-proxy" Oct 29 23:31:00.937148 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.937118 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" Oct 29 23:31:00.939010 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.938976 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr"] Oct 29 23:31:00.951148 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.951122 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:31:00.951283 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.951219 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:31:00.951346 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.951288 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:31:00.951406 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:00.951365 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:31:01.073674 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:01.073622 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963cedd3-0342-431f-939b-627bd09b94c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr\" (UID: \"963cedd3-0342-431f-939b-627bd09b94c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" Oct 29 23:31:01.073864 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:01.073736 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hfht\" (UniqueName: \"kubernetes.io/projected/963cedd3-0342-431f-939b-627bd09b94c4-kube-api-access-7hfht\") pod \"maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr\" (UID: \"963cedd3-0342-431f-939b-627bd09b94c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" Oct 29 23:31:01.073978 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:01.073960 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963cedd3-0342-431f-939b-627bd09b94c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr\" (UID: \"963cedd3-0342-431f-939b-627bd09b94c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" Oct 29 23:31:01.175083 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:01.175047 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963cedd3-0342-431f-939b-627bd09b94c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr\" (UID: \"963cedd3-0342-431f-939b-627bd09b94c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" Oct 29 23:31:01.175274 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:01.175097 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hfht\" (UniqueName: \"kubernetes.io/projected/963cedd3-0342-431f-939b-627bd09b94c4-kube-api-access-7hfht\") pod \"maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr\" (UID: \"963cedd3-0342-431f-939b-627bd09b94c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" Oct 29 23:31:01.175274 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:01.175182 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963cedd3-0342-431f-939b-627bd09b94c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr\" (UID: \"963cedd3-0342-431f-939b-627bd09b94c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" Oct 29 23:31:01.175800 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:01.175770 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963cedd3-0342-431f-939b-627bd09b94c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr\" (UID: \"963cedd3-0342-431f-939b-627bd09b94c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" Oct 29 23:31:01.177426 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:01.177405 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963cedd3-0342-431f-939b-627bd09b94c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr\" (UID: \"963cedd3-0342-431f-939b-627bd09b94c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" Oct 29 23:31:01.192935 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:01.192908 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hfht\" (UniqueName: \"kubernetes.io/projected/963cedd3-0342-431f-939b-627bd09b94c4-kube-api-access-7hfht\") pod \"maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr\" (UID: \"963cedd3-0342-431f-939b-627bd09b94c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" Oct 29 23:31:01.266791 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:01.266761 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" Oct 29 23:31:01.526857 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:01.524861 6507 patch_prober.go:29] interesting pod/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 29 23:31:01.526857 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:01.524927 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" podUID="80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 29 23:31:01.591801 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:01.591719 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr"] Oct 29 23:31:01.594284 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:31:01.594249 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963cedd3_0342_431f_939b_627bd09b94c4.slice/crio-606905ba8928816cf28bf6e2c1f86866da7aebe54f31461620ded6b34236cb43 WatchSource:0}: Error finding container 606905ba8928816cf28bf6e2c1f86866da7aebe54f31461620ded6b34236cb43: Status 404 returned error can't find the container with id 606905ba8928816cf28bf6e2c1f86866da7aebe54f31461620ded6b34236cb43 Oct 29 23:31:02.436134 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:02.436100 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" event={"ID":"963cedd3-0342-431f-939b-627bd09b94c4","Type":"ContainerStarted","Data":"545c963711ad13d69539d7488d754cda6476d444fba84e89d79cd8bc5976bd85"} Oct 29 23:31:02.436287 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:02.436153 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" event={"ID":"963cedd3-0342-431f-939b-627bd09b94c4","Type":"ContainerStarted","Data":"606905ba8928816cf28bf6e2c1f86866da7aebe54f31461620ded6b34236cb43"} Oct 29 23:31:02.473679 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:02.473606 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" podStartSLOduration=2.473584529 podStartE2EDuration="2.473584529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:02.469687347 +0000 UTC m=+1031.832692070" watchObservedRunningTime="2026-10-29 23:31:02.473584529 +0000 UTC m=+1031.836589253" Oct 29 23:31:02.766190 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:02.766159 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:31:02.767480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:02.767425 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:31:02.768161 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:02.768126 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:31:03.445684 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:03.445654 6507 generic.go:356] "Generic (PLEG): container finished" podID="963cedd3-0342-431f-939b-627bd09b94c4" containerID="545c963711ad13d69539d7488d754cda6476d444fba84e89d79cd8bc5976bd85" exitCode=0 Oct 29 23:31:03.445846 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:03.445741 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" event={"ID":"963cedd3-0342-431f-939b-627bd09b94c4","Type":"ContainerDied","Data":"545c963711ad13d69539d7488d754cda6476d444fba84e89d79cd8bc5976bd85"} Oct 29 23:31:04.404731 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:04.404693 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:31:04.602833 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:04.602808 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" Oct 29 23:31:04.721511 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:04.721473 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963cedd3-0342-431f-939b-627bd09b94c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963cedd3-0342-431f-939b-627bd09b94c4-maas-api-openshift-ca\") pod \"963cedd3-0342-431f-939b-627bd09b94c4\" (UID: \"963cedd3-0342-431f-939b-627bd09b94c4\") " Oct 29 23:31:04.721511 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:04.721519 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963cedd3-0342-431f-939b-627bd09b94c4-kube-api-access-7hfht\" (UniqueName: \"kubernetes.io/projected/963cedd3-0342-431f-939b-627bd09b94c4-kube-api-access-7hfht\") pod \"963cedd3-0342-431f-939b-627bd09b94c4\" (UID: \"963cedd3-0342-431f-939b-627bd09b94c4\") " Oct 29 23:31:04.721712 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:04.721656 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/963cedd3-0342-431f-939b-627bd09b94c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963cedd3-0342-431f-939b-627bd09b94c4-maas-api-xks-ca\") pod \"963cedd3-0342-431f-939b-627bd09b94c4\" (UID: \"963cedd3-0342-431f-939b-627bd09b94c4\") " Oct 29 23:31:04.721899 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:04.721874 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963cedd3-0342-431f-939b-627bd09b94c4-maas-api-openshift-ca" pod "963cedd3-0342-431f-939b-627bd09b94c4" (UID: "963cedd3-0342-431f-939b-627bd09b94c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:04.723654 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:04.723627 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963cedd3-0342-431f-939b-627bd09b94c4-maas-api-xks-ca" pod "963cedd3-0342-431f-939b-627bd09b94c4" (UID: "963cedd3-0342-431f-939b-627bd09b94c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:04.723750 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:04.723727 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963cedd3-0342-431f-939b-627bd09b94c4-kube-api-access-7hfht" pod "963cedd3-0342-431f-939b-627bd09b94c4" (UID: "963cedd3-0342-431f-939b-627bd09b94c4"). InnerVolumeSpecName "kube-api-access-7hfht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:04.822966 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:04.822931 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963cedd3-0342-431f-939b-627bd09b94c4-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:04.822966 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:04.822963 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963cedd3-0342-431f-939b-627bd09b94c4-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:04.822966 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:04.822973 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hfht\" (UniqueName: \"kubernetes.io/projected/963cedd3-0342-431f-939b-627bd09b94c4-kube-api-access-7hfht\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:05.461082 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:05.461052 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" Oct 29 23:31:05.461082 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:05.461062 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr" event={"ID":"963cedd3-0342-431f-939b-627bd09b94c4","Type":"ContainerDied","Data":"606905ba8928816cf28bf6e2c1f86866da7aebe54f31461620ded6b34236cb43"} Oct 29 23:31:05.461640 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:05.461107 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="606905ba8928816cf28bf6e2c1f86866da7aebe54f31461620ded6b34236cb43" Oct 29 23:31:05.705154 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:05.705113 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz"] Oct 29 23:31:05.710881 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:05.710849 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m"] Oct 29 23:31:09.077470 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:09.077431 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8"] Oct 29 23:31:09.090268 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:09.090237 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df53e62-8t8g8"] Oct 29 23:31:10.739528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.739491 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8"] Oct 29 23:31:10.739973 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.739954 6507 cpu_manager.go:414] "containerMap: removing container" podUID="963cedd3-0342-431f-939b-627bd09b94c4" containerName="revoke-keys" Oct 29 23:31:10.739973 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.739970 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963cedd3-0342-431f-939b-627bd09b94c4" containerName="revoke-keys" Oct 29 23:31:10.740129 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.740076 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="963cedd3-0342-431f-939b-627bd09b94c4" containerName="revoke-keys" Oct 29 23:31:10.740129 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.740085 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa280316-fb6c-40db-8658-65b84bb4382b" containerName="istio-proxy" Oct 29 23:31:10.743364 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.743346 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8" Oct 29 23:31:10.753760 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.753728 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6093b1-1a67-46ef-b324-c9433324fb9c" path="/var/lib/kubelet/pods/2b6093b1-1a67-46ef-b324-c9433324fb9c/volumes" Oct 29 23:31:10.765082 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.765049 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:31:10.765309 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.765090 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:31:10.765309 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.765103 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:31:10.765472 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.765119 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:31:10.768698 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.768672 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8"] Oct 29 23:31:10.806195 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.806161 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb8zh\" (UniqueName: \"kubernetes.io/projected/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-kube-api-access-qb8zh\") pod \"maas-api-revoke-sub-99d886a5-v2cw8\" (UID: \"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8" Oct 29 23:31:10.806378 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.806205 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99d886a5-v2cw8\" (UID: \"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8" Oct 29 23:31:10.806378 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.806347 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99d886a5-v2cw8\" (UID: \"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8" Oct 29 23:31:10.907383 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.907347 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb8zh\" (UniqueName: \"kubernetes.io/projected/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-kube-api-access-qb8zh\") pod \"maas-api-revoke-sub-99d886a5-v2cw8\" (UID: \"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8" Oct 29 23:31:10.907383 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.907389 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99d886a5-v2cw8\" (UID: \"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8" Oct 29 23:31:10.907666 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.907450 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99d886a5-v2cw8\" (UID: \"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8" Oct 29 23:31:10.908053 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.908028 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99d886a5-v2cw8\" (UID: \"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8" Oct 29 23:31:10.909775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.909757 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99d886a5-v2cw8\" (UID: \"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8" Oct 29 23:31:10.924432 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:10.924407 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb8zh\" (UniqueName: \"kubernetes.io/projected/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-kube-api-access-qb8zh\") pod \"maas-api-revoke-sub-99d886a5-v2cw8\" (UID: \"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8" Oct 29 23:31:11.057897 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:11.057808 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8" Oct 29 23:31:11.359165 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:11.359136 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8"] Oct 29 23:31:11.361349 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:31:11.361320 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8623ac86_e80b_4c9c_bc3d_73c5ef5b70f9.slice/crio-4d8d7eae63eca5db951a024037ad428d0ec7890511cd2eea26c4537750c11428 WatchSource:0}: Error finding container 4d8d7eae63eca5db951a024037ad428d0ec7890511cd2eea26c4537750c11428: Status 404 returned error can't find the container with id 4d8d7eae63eca5db951a024037ad428d0ec7890511cd2eea26c4537750c11428 Oct 29 23:31:11.505118 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:11.505066 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8" event={"ID":"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9","Type":"ContainerStarted","Data":"4d8d7eae63eca5db951a024037ad428d0ec7890511cd2eea26c4537750c11428"} Oct 29 23:31:12.559474 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:12.559363 6507 generic.go:356] "Generic (PLEG): container finished" podID="8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9" containerID="2e6e5cdbc87158206f0119b424435c206ed920a6543f248c16d904857c075108" exitCode=0 Oct 29 23:31:12.559474 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:12.559408 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8" event={"ID":"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9","Type":"ContainerDied","Data":"2e6e5cdbc87158206f0119b424435c206ed920a6543f248c16d904857c075108"} Oct 29 23:31:13.706576 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.706538 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r"] Oct 29 23:31:13.710635 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.710611 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r" Oct 29 23:31:13.735693 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.731979 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r"] Oct 29 23:31:13.751361 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.751333 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8" Oct 29 23:31:13.761872 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.761831 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175a760e-0d42-42e9-a626-890e1cb9bc27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4832c42-bv59r\" (UID: \"175a760e-0d42-42e9-a626-890e1cb9bc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r" Oct 29 23:31:13.762082 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.761992 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175a760e-0d42-42e9-a626-890e1cb9bc27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4832c42-bv59r\" (UID: \"175a760e-0d42-42e9-a626-890e1cb9bc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r" Oct 29 23:31:13.762082 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.762024 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxfpd\" (UniqueName: \"kubernetes.io/projected/175a760e-0d42-42e9-a626-890e1cb9bc27-kube-api-access-rxfpd\") pod \"maas-api-revoke-sub-c4832c42-bv59r\" (UID: \"175a760e-0d42-42e9-a626-890e1cb9bc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r" Oct 29 23:31:13.862861 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.862774 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-maas-api-xks-ca\") pod \"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9\" (UID: \"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9\") " Oct 29 23:31:13.863019 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.862941 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-maas-api-openshift-ca\") pod \"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9\" (UID: \"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9\") " Oct 29 23:31:13.863070 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.863053 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-kube-api-access-qb8zh\" (UniqueName: \"kubernetes.io/projected/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-kube-api-access-qb8zh\") pod \"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9\" (UID: \"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9\") " Oct 29 23:31:13.863309 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.863283 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-maas-api-openshift-ca" pod "8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9" (UID: "8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:13.863425 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.863377 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175a760e-0d42-42e9-a626-890e1cb9bc27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4832c42-bv59r\" (UID: \"175a760e-0d42-42e9-a626-890e1cb9bc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r" Oct 29 23:31:13.863425 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.863407 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxfpd\" (UniqueName: \"kubernetes.io/projected/175a760e-0d42-42e9-a626-890e1cb9bc27-kube-api-access-rxfpd\") pod \"maas-api-revoke-sub-c4832c42-bv59r\" (UID: \"175a760e-0d42-42e9-a626-890e1cb9bc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r" Oct 29 23:31:13.863534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.863520 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175a760e-0d42-42e9-a626-890e1cb9bc27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4832c42-bv59r\" (UID: \"175a760e-0d42-42e9-a626-890e1cb9bc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r" Oct 29 23:31:13.863873 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.863695 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:13.864329 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.864308 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175a760e-0d42-42e9-a626-890e1cb9bc27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4832c42-bv59r\" (UID: \"175a760e-0d42-42e9-a626-890e1cb9bc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r" Oct 29 23:31:13.865278 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.865252 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-kube-api-access-qb8zh" pod "8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9" (UID: "8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9"). InnerVolumeSpecName "kube-api-access-qb8zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:13.865562 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.865541 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-maas-api-xks-ca" pod "8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9" (UID: "8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:13.865944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.865927 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175a760e-0d42-42e9-a626-890e1cb9bc27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4832c42-bv59r\" (UID: \"175a760e-0d42-42e9-a626-890e1cb9bc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r" Oct 29 23:31:13.880435 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.880400 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxfpd\" (UniqueName: \"kubernetes.io/projected/175a760e-0d42-42e9-a626-890e1cb9bc27-kube-api-access-rxfpd\") pod \"maas-api-revoke-sub-c4832c42-bv59r\" (UID: \"175a760e-0d42-42e9-a626-890e1cb9bc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r" Oct 29 23:31:13.965183 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.965149 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb8zh\" (UniqueName: \"kubernetes.io/projected/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-kube-api-access-qb8zh\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:13.965183 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:13.965182 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:14.060201 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:14.059664 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r" Oct 29 23:31:14.365903 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:14.365873 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r"] Oct 29 23:31:14.368358 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:31:14.368330 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175a760e_0d42_42e9_a626_890e1cb9bc27.slice/crio-32569c0011f882f1e3c61a7780cdeb6b1e9551a48f128ae8850b3aa800793594 WatchSource:0}: Error finding container 32569c0011f882f1e3c61a7780cdeb6b1e9551a48f128ae8850b3aa800793594: Status 404 returned error can't find the container with id 32569c0011f882f1e3c61a7780cdeb6b1e9551a48f128ae8850b3aa800793594 Oct 29 23:31:14.574678 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:14.574639 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8" Oct 29 23:31:14.574678 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:14.574669 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8" event={"ID":"8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9","Type":"ContainerDied","Data":"4d8d7eae63eca5db951a024037ad428d0ec7890511cd2eea26c4537750c11428"} Oct 29 23:31:14.574937 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:14.574724 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d8d7eae63eca5db951a024037ad428d0ec7890511cd2eea26c4537750c11428" Oct 29 23:31:14.576113 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:14.576083 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r" event={"ID":"175a760e-0d42-42e9-a626-890e1cb9bc27","Type":"ContainerStarted","Data":"32569c0011f882f1e3c61a7780cdeb6b1e9551a48f128ae8850b3aa800793594"} Oct 29 23:31:15.225778 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:31:15.225739 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:31:15.607229 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:15.607135 6507 generic.go:356] "Generic (PLEG): container finished" podID="175a760e-0d42-42e9-a626-890e1cb9bc27" containerID="4157c968937e4488e68d8d9174aa9009eb01fede7cd8e84c1d7a4f4ed66e557f" exitCode=0 Oct 29 23:31:15.607393 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:15.607282 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r" event={"ID":"175a760e-0d42-42e9-a626-890e1cb9bc27","Type":"ContainerDied","Data":"4157c968937e4488e68d8d9174aa9009eb01fede7cd8e84c1d7a4f4ed66e557f"} Oct 29 23:31:16.078192 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:16.078154 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:31:16.779582 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:16.779550 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r" Oct 29 23:31:16.812958 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:16.812911 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/175a760e-0d42-42e9-a626-890e1cb9bc27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175a760e-0d42-42e9-a626-890e1cb9bc27-maas-api-xks-ca\") pod \"175a760e-0d42-42e9-a626-890e1cb9bc27\" (UID: \"175a760e-0d42-42e9-a626-890e1cb9bc27\") " Oct 29 23:31:16.813150 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:16.813117 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/175a760e-0d42-42e9-a626-890e1cb9bc27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175a760e-0d42-42e9-a626-890e1cb9bc27-maas-api-openshift-ca\") pod \"175a760e-0d42-42e9-a626-890e1cb9bc27\" (UID: \"175a760e-0d42-42e9-a626-890e1cb9bc27\") " Oct 29 23:31:16.813231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:16.813215 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175a760e-0d42-42e9-a626-890e1cb9bc27-kube-api-access-rxfpd\" (UniqueName: \"kubernetes.io/projected/175a760e-0d42-42e9-a626-890e1cb9bc27-kube-api-access-rxfpd\") pod \"175a760e-0d42-42e9-a626-890e1cb9bc27\" (UID: \"175a760e-0d42-42e9-a626-890e1cb9bc27\") " Oct 29 23:31:16.813529 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:16.813502 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175a760e-0d42-42e9-a626-890e1cb9bc27-maas-api-openshift-ca" pod "175a760e-0d42-42e9-a626-890e1cb9bc27" (UID: "175a760e-0d42-42e9-a626-890e1cb9bc27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:16.814506 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:16.813860 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175a760e-0d42-42e9-a626-890e1cb9bc27-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:16.816096 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:16.815792 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/175a760e-0d42-42e9-a626-890e1cb9bc27-maas-api-xks-ca" pod "175a760e-0d42-42e9-a626-890e1cb9bc27" (UID: "175a760e-0d42-42e9-a626-890e1cb9bc27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:16.816096 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:16.815812 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175a760e-0d42-42e9-a626-890e1cb9bc27-kube-api-access-rxfpd" pod "175a760e-0d42-42e9-a626-890e1cb9bc27" (UID: "175a760e-0d42-42e9-a626-890e1cb9bc27"). InnerVolumeSpecName "kube-api-access-rxfpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:16.914921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:16.914818 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxfpd\" (UniqueName: \"kubernetes.io/projected/175a760e-0d42-42e9-a626-890e1cb9bc27-kube-api-access-rxfpd\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:16.914921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:16.914864 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175a760e-0d42-42e9-a626-890e1cb9bc27-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:17.625435 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:17.625399 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r" Oct 29 23:31:17.625652 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:17.625431 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r" event={"ID":"175a760e-0d42-42e9-a626-890e1cb9bc27","Type":"ContainerDied","Data":"32569c0011f882f1e3c61a7780cdeb6b1e9551a48f128ae8850b3aa800793594"} Oct 29 23:31:17.625652 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:17.625506 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32569c0011f882f1e3c61a7780cdeb6b1e9551a48f128ae8850b3aa800793594" Oct 29 23:31:20.726011 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:20.725968 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" containerID="cri-o://ea7e85606c419a74ba75728d986f8818442d08c11c3712ca725a9c2ba2d410bc" gracePeriod=45 Oct 29 23:31:20.731017 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:20.730972 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" containerID="cri-o://3d2c97d359c86ed2b7c369b97c94546f9d33db0bdc20bf2781591f2ec77f47f9" gracePeriod=45 Oct 29 23:31:20.774818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:20.774778 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": dial tcp 10.133.0.78:8000: connect: connection refused" Oct 29 23:31:21.080639 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.076028 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:31:21.090530 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.081664 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:31:21.170749 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.170335 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da92dda9-c888-40e8-8790-7d9fc8a86ef7-kube-api-access-k9rwh\" (UniqueName: \"kubernetes.io/projected/da92dda9-c888-40e8-8790-7d9fc8a86ef7-kube-api-access-k9rwh\") pod \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " Oct 29 23:31:21.170749 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.170395 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-tmp-dir\") pod \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " Oct 29 23:31:21.170749 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.170447 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da92dda9-c888-40e8-8790-7d9fc8a86ef7-tls-certs\" (UniqueName: \"kubernetes.io/secret/da92dda9-c888-40e8-8790-7d9fc8a86ef7-tls-certs\") pod \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " Oct 29 23:31:21.170749 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.170551 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-dshm\") pod \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " Oct 29 23:31:21.170749 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.170587 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-dshm\") pod \"f53490ca-3cd8-425e-adfd-cf31eec83280\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " Oct 29 23:31:21.170749 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.170619 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f53490ca-3cd8-425e-adfd-cf31eec83280-tls-certs\" (UniqueName: \"kubernetes.io/secret/f53490ca-3cd8-425e-adfd-cf31eec83280-tls-certs\") pod \"f53490ca-3cd8-425e-adfd-cf31eec83280\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " Oct 29 23:31:21.170749 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.170678 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-model-cache\") pod \"f53490ca-3cd8-425e-adfd-cf31eec83280\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " Oct 29 23:31:21.170749 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.170715 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-tmp-dir\") pod \"f53490ca-3cd8-425e-adfd-cf31eec83280\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " Oct 29 23:31:21.171311 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.170779 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53490ca-3cd8-425e-adfd-cf31eec83280-kube-api-access-4z8g2\" (UniqueName: \"kubernetes.io/projected/f53490ca-3cd8-425e-adfd-cf31eec83280-kube-api-access-4z8g2\") pod \"f53490ca-3cd8-425e-adfd-cf31eec83280\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " Oct 29 23:31:21.171311 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.170821 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-model-cache\") pod \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " Oct 29 23:31:21.171311 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.170877 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-home\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-home\") pod \"f53490ca-3cd8-425e-adfd-cf31eec83280\" (UID: \"f53490ca-3cd8-425e-adfd-cf31eec83280\") " Oct 29 23:31:21.171311 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.170947 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-home\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-home\") pod \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\" (UID: \"da92dda9-c888-40e8-8790-7d9fc8a86ef7\") " Oct 29 23:31:21.171311 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.171170 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-tmp-dir" pod "da92dda9-c888-40e8-8790-7d9fc8a86ef7" (UID: "da92dda9-c888-40e8-8790-7d9fc8a86ef7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:21.171607 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.171304 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-tmp-dir" pod "f53490ca-3cd8-425e-adfd-cf31eec83280" (UID: "f53490ca-3cd8-425e-adfd-cf31eec83280"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:21.171607 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.171426 6507 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-tmp-dir\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:21.171607 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.171451 6507 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-tmp-dir\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:21.171607 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.171497 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-model-cache" pod "f53490ca-3cd8-425e-adfd-cf31eec83280" (UID: "f53490ca-3cd8-425e-adfd-cf31eec83280"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:21.172091 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.172062 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-home" pod "da92dda9-c888-40e8-8790-7d9fc8a86ef7" (UID: "da92dda9-c888-40e8-8790-7d9fc8a86ef7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:21.172588 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.172565 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-home" pod "f53490ca-3cd8-425e-adfd-cf31eec83280" (UID: "f53490ca-3cd8-425e-adfd-cf31eec83280"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:21.174144 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.174105 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-model-cache" pod "da92dda9-c888-40e8-8790-7d9fc8a86ef7" (UID: "da92dda9-c888-40e8-8790-7d9fc8a86ef7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:21.178524 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.178495 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-dshm" pod "f53490ca-3cd8-425e-adfd-cf31eec83280" (UID: "f53490ca-3cd8-425e-adfd-cf31eec83280"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:21.178970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.178803 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53490ca-3cd8-425e-adfd-cf31eec83280-kube-api-access-4z8g2" pod "f53490ca-3cd8-425e-adfd-cf31eec83280" (UID: "f53490ca-3cd8-425e-adfd-cf31eec83280"). InnerVolumeSpecName "kube-api-access-4z8g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:21.179276 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.179239 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da92dda9-c888-40e8-8790-7d9fc8a86ef7-tls-certs" pod "da92dda9-c888-40e8-8790-7d9fc8a86ef7" (UID: "da92dda9-c888-40e8-8790-7d9fc8a86ef7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:21.179559 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.179529 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da92dda9-c888-40e8-8790-7d9fc8a86ef7-kube-api-access-k9rwh" pod "da92dda9-c888-40e8-8790-7d9fc8a86ef7" (UID: "da92dda9-c888-40e8-8790-7d9fc8a86ef7"). InnerVolumeSpecName "kube-api-access-k9rwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:21.179643 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.179586 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-dshm" pod "da92dda9-c888-40e8-8790-7d9fc8a86ef7" (UID: "da92dda9-c888-40e8-8790-7d9fc8a86ef7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:21.179752 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.179725 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f53490ca-3cd8-425e-adfd-cf31eec83280-tls-certs" pod "f53490ca-3cd8-425e-adfd-cf31eec83280" (UID: "f53490ca-3cd8-425e-adfd-cf31eec83280"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:21.272589 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.272501 6507 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-home\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:21.272589 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.272537 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9rwh\" (UniqueName: \"kubernetes.io/projected/da92dda9-c888-40e8-8790-7d9fc8a86ef7-kube-api-access-k9rwh\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:21.272589 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.272549 6507 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da92dda9-c888-40e8-8790-7d9fc8a86ef7-tls-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:21.272589 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.272559 6507 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-dshm\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:21.272589 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.272566 6507 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-dshm\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:21.272589 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.272574 6507 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53490ca-3cd8-425e-adfd-cf31eec83280-tls-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:21.272589 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.272582 6507 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-model-cache\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:21.272589 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.272590 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z8g2\" (UniqueName: \"kubernetes.io/projected/f53490ca-3cd8-425e-adfd-cf31eec83280-kube-api-access-4z8g2\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:21.272940 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.272598 6507 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da92dda9-c888-40e8-8790-7d9fc8a86ef7-model-cache\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:21.272940 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.272607 6507 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53490ca-3cd8-425e-adfd-cf31eec83280-home\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:21.656819 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.656725 6507 generic.go:356] "Generic (PLEG): container finished" podID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerID="3d2c97d359c86ed2b7c369b97c94546f9d33db0bdc20bf2781591f2ec77f47f9" exitCode=0 Oct 29 23:31:21.656819 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.656792 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" Oct 29 23:31:21.657065 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.656814 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" event={"ID":"da92dda9-c888-40e8-8790-7d9fc8a86ef7","Type":"ContainerDied","Data":"3d2c97d359c86ed2b7c369b97c94546f9d33db0bdc20bf2781591f2ec77f47f9"} Oct 29 23:31:21.657065 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.656874 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m" event={"ID":"da92dda9-c888-40e8-8790-7d9fc8a86ef7","Type":"ContainerDied","Data":"ca6dd7e8833d69185d18bf1c543381f4fa44109af77a6985c5edbf9b5926bbe1"} Oct 29 23:31:21.657065 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.656899 6507 scope.go:122] "RemoveContainer" containerID="3d2c97d359c86ed2b7c369b97c94546f9d33db0bdc20bf2781591f2ec77f47f9" Oct 29 23:31:21.659108 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.659078 6507 generic.go:356] "Generic (PLEG): container finished" podID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerID="ea7e85606c419a74ba75728d986f8818442d08c11c3712ca725a9c2ba2d410bc" exitCode=0 Oct 29 23:31:21.659288 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.659168 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" Oct 29 23:31:21.659288 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.659215 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" event={"ID":"f53490ca-3cd8-425e-adfd-cf31eec83280","Type":"ContainerDied","Data":"ea7e85606c419a74ba75728d986f8818442d08c11c3712ca725a9c2ba2d410bc"} Oct 29 23:31:21.659288 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.659245 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz" event={"ID":"f53490ca-3cd8-425e-adfd-cf31eec83280","Type":"ContainerDied","Data":"139d119024c790204866e951586d6c08dfa361eef423ad5e14821ac9a71feceb"} Oct 29 23:31:21.670965 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.670946 6507 scope.go:122] "RemoveContainer" containerID="3d2c97d359c86ed2b7c369b97c94546f9d33db0bdc20bf2781591f2ec77f47f9" Oct 29 23:31:21.671268 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:31:21.671235 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2c97d359c86ed2b7c369b97c94546f9d33db0bdc20bf2781591f2ec77f47f9\": container with ID starting with 3d2c97d359c86ed2b7c369b97c94546f9d33db0bdc20bf2781591f2ec77f47f9 not found: ID does not exist" containerID="3d2c97d359c86ed2b7c369b97c94546f9d33db0bdc20bf2781591f2ec77f47f9" Oct 29 23:31:21.671268 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.671260 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2c97d359c86ed2b7c369b97c94546f9d33db0bdc20bf2781591f2ec77f47f9"} err="failed to get container status \"3d2c97d359c86ed2b7c369b97c94546f9d33db0bdc20bf2781591f2ec77f47f9\": rpc error: code = NotFound desc = could not find container \"3d2c97d359c86ed2b7c369b97c94546f9d33db0bdc20bf2781591f2ec77f47f9\": container with ID starting with 3d2c97d359c86ed2b7c369b97c94546f9d33db0bdc20bf2781591f2ec77f47f9 not found: ID does not exist" Oct 29 23:31:21.671424 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.671276 6507 scope.go:122] "RemoveContainer" containerID="ea7e85606c419a74ba75728d986f8818442d08c11c3712ca725a9c2ba2d410bc" Oct 29 23:31:21.680186 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.680167 6507 scope.go:122] "RemoveContainer" containerID="ea7e85606c419a74ba75728d986f8818442d08c11c3712ca725a9c2ba2d410bc" Oct 29 23:31:21.682774 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:31:21.682741 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7e85606c419a74ba75728d986f8818442d08c11c3712ca725a9c2ba2d410bc\": container with ID starting with ea7e85606c419a74ba75728d986f8818442d08c11c3712ca725a9c2ba2d410bc not found: ID does not exist" containerID="ea7e85606c419a74ba75728d986f8818442d08c11c3712ca725a9c2ba2d410bc" Oct 29 23:31:21.682861 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.682780 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7e85606c419a74ba75728d986f8818442d08c11c3712ca725a9c2ba2d410bc"} err="failed to get container status \"ea7e85606c419a74ba75728d986f8818442d08c11c3712ca725a9c2ba2d410bc\": rpc error: code = NotFound desc = could not find container \"ea7e85606c419a74ba75728d986f8818442d08c11c3712ca725a9c2ba2d410bc\": container with ID starting with ea7e85606c419a74ba75728d986f8818442d08c11c3712ca725a9c2ba2d410bc not found: ID does not exist" Oct 29 23:31:21.715090 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.715059 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m"] Oct 29 23:31:21.720637 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.720604 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-be8726/distinct-w2-be8726-kserve-56879b868c-lvf8m"] Oct 29 23:31:21.754470 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.754413 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz"] Oct 29 23:31:21.758005 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.757973 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-be8726/distinct-2-w2-be8726-kserve-58d64bbff6-vz5vz"] Oct 29 23:31:21.779063 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.779030 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj"] Oct 29 23:31:21.779534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.779519 6507 cpu_manager.go:414] "containerMap: removing container" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" Oct 29 23:31:21.779592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.779535 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" Oct 29 23:31:21.779592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.779547 6507 cpu_manager.go:414] "containerMap: removing container" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" Oct 29 23:31:21.779592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.779553 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" Oct 29 23:31:21.779592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.779571 6507 cpu_manager.go:414] "containerMap: removing container" podUID="175a760e-0d42-42e9-a626-890e1cb9bc27" containerName="revoke-keys" Oct 29 23:31:21.779592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.779576 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175a760e-0d42-42e9-a626-890e1cb9bc27" containerName="revoke-keys" Oct 29 23:31:21.779592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.779590 6507 cpu_manager.go:414] "containerMap: removing container" podUID="8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9" containerName="revoke-keys" Oct 29 23:31:21.779759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.779596 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9" containerName="revoke-keys" Oct 29 23:31:21.779759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.779675 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="175a760e-0d42-42e9-a626-890e1cb9bc27" containerName="revoke-keys" Oct 29 23:31:21.779759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.779686 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9" containerName="revoke-keys" Oct 29 23:31:21.779759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.779696 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" containerName="main" Oct 29 23:31:21.779759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.779702 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" containerName="main" Oct 29 23:31:21.784695 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.784672 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj" Oct 29 23:31:21.797397 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.797373 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:31:21.797578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.797411 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:31:21.797578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.797412 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:31:21.797578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.797467 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:31:21.801366 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.801339 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj"] Oct 29 23:31:21.882761 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.882727 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706e7030-0c86-4612-9238-4a3e1b0075b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj\" (UID: \"706e7030-0c86-4612-9238-4a3e1b0075b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj" Oct 29 23:31:21.882941 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.882807 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bcld\" (UniqueName: \"kubernetes.io/projected/706e7030-0c86-4612-9238-4a3e1b0075b7-kube-api-access-9bcld\") pod \"maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj\" (UID: \"706e7030-0c86-4612-9238-4a3e1b0075b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj" Oct 29 23:31:21.882941 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.882891 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706e7030-0c86-4612-9238-4a3e1b0075b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj\" (UID: \"706e7030-0c86-4612-9238-4a3e1b0075b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj" Oct 29 23:31:21.984513 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.984473 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706e7030-0c86-4612-9238-4a3e1b0075b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj\" (UID: \"706e7030-0c86-4612-9238-4a3e1b0075b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj" Oct 29 23:31:21.984671 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.984593 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706e7030-0c86-4612-9238-4a3e1b0075b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj\" (UID: \"706e7030-0c86-4612-9238-4a3e1b0075b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj" Oct 29 23:31:21.984671 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.984618 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bcld\" (UniqueName: \"kubernetes.io/projected/706e7030-0c86-4612-9238-4a3e1b0075b7-kube-api-access-9bcld\") pod \"maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj\" (UID: \"706e7030-0c86-4612-9238-4a3e1b0075b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj" Oct 29 23:31:21.985188 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.985167 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706e7030-0c86-4612-9238-4a3e1b0075b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj\" (UID: \"706e7030-0c86-4612-9238-4a3e1b0075b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj" Oct 29 23:31:21.987084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:21.987057 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706e7030-0c86-4612-9238-4a3e1b0075b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj\" (UID: \"706e7030-0c86-4612-9238-4a3e1b0075b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj" Oct 29 23:31:22.004037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:22.004007 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bcld\" (UniqueName: \"kubernetes.io/projected/706e7030-0c86-4612-9238-4a3e1b0075b7-kube-api-access-9bcld\") pod \"maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj\" (UID: \"706e7030-0c86-4612-9238-4a3e1b0075b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj" Oct 29 23:31:22.098134 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:22.098098 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj" Oct 29 23:31:22.400806 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:22.400762 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj"] Oct 29 23:31:22.403245 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:31:22.403222 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706e7030_0c86_4612_9238_4a3e1b0075b7.slice/crio-ffefec53f652c3f1a7b84aa4b5335da409a329e89d26c17eeb7ae1133d841446 WatchSource:0}: Error finding container ffefec53f652c3f1a7b84aa4b5335da409a329e89d26c17eeb7ae1133d841446: Status 404 returned error can't find the container with id ffefec53f652c3f1a7b84aa4b5335da409a329e89d26c17eeb7ae1133d841446 Oct 29 23:31:22.666760 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:22.666647 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj" event={"ID":"706e7030-0c86-4612-9238-4a3e1b0075b7","Type":"ContainerStarted","Data":"ffefec53f652c3f1a7b84aa4b5335da409a329e89d26c17eeb7ae1133d841446"} Oct 29 23:31:22.752968 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:22.752935 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da92dda9-c888-40e8-8790-7d9fc8a86ef7" path="/var/lib/kubelet/pods/da92dda9-c888-40e8-8790-7d9fc8a86ef7/volumes" Oct 29 23:31:22.753434 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:22.753417 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53490ca-3cd8-425e-adfd-cf31eec83280" path="/var/lib/kubelet/pods/f53490ca-3cd8-425e-adfd-cf31eec83280/volumes" Oct 29 23:31:23.055481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:23.050520 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9"] Oct 29 23:31:23.060264 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:23.060128 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0989d7c4-sl5d9"] Oct 29 23:31:23.676432 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:23.676394 6507 generic.go:356] "Generic (PLEG): container finished" podID="706e7030-0c86-4612-9238-4a3e1b0075b7" containerID="707ad2992e44631615d2558222c89eca368b466dc76a9034f9eb42118c2271c7" exitCode=0 Oct 29 23:31:23.676631 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:23.676505 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj" event={"ID":"706e7030-0c86-4612-9238-4a3e1b0075b7","Type":"ContainerDied","Data":"707ad2992e44631615d2558222c89eca368b466dc76a9034f9eb42118c2271c7"} Oct 29 23:31:24.753843 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:24.753803 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41db8ac2-ac01-48ce-927e-51d6f6e1eaa3" path="/var/lib/kubelet/pods/41db8ac2-ac01-48ce-927e-51d6f6e1eaa3/volumes" Oct 29 23:31:24.817195 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:24.817167 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj" Oct 29 23:31:24.887774 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:24.887715 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr"] Oct 29 23:31:24.892563 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:24.892530 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-be8726-lk8dr"] Oct 29 23:31:24.924903 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:24.924871 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/706e7030-0c86-4612-9238-4a3e1b0075b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706e7030-0c86-4612-9238-4a3e1b0075b7-maas-api-openshift-ca\") pod \"706e7030-0c86-4612-9238-4a3e1b0075b7\" (UID: \"706e7030-0c86-4612-9238-4a3e1b0075b7\") " Oct 29 23:31:24.925070 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:24.924922 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706e7030-0c86-4612-9238-4a3e1b0075b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706e7030-0c86-4612-9238-4a3e1b0075b7-maas-api-xks-ca\") pod \"706e7030-0c86-4612-9238-4a3e1b0075b7\" (UID: \"706e7030-0c86-4612-9238-4a3e1b0075b7\") " Oct 29 23:31:24.925070 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:24.925010 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706e7030-0c86-4612-9238-4a3e1b0075b7-kube-api-access-9bcld\" (UniqueName: \"kubernetes.io/projected/706e7030-0c86-4612-9238-4a3e1b0075b7-kube-api-access-9bcld\") pod \"706e7030-0c86-4612-9238-4a3e1b0075b7\" (UID: \"706e7030-0c86-4612-9238-4a3e1b0075b7\") " Oct 29 23:31:24.925205 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:24.925182 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/706e7030-0c86-4612-9238-4a3e1b0075b7-maas-api-openshift-ca" pod "706e7030-0c86-4612-9238-4a3e1b0075b7" (UID: "706e7030-0c86-4612-9238-4a3e1b0075b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:24.925374 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:24.925356 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/706e7030-0c86-4612-9238-4a3e1b0075b7-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:24.927110 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:24.927080 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706e7030-0c86-4612-9238-4a3e1b0075b7-maas-api-xks-ca" pod "706e7030-0c86-4612-9238-4a3e1b0075b7" (UID: "706e7030-0c86-4612-9238-4a3e1b0075b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:24.927110 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:24.927097 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706e7030-0c86-4612-9238-4a3e1b0075b7-kube-api-access-9bcld" pod "706e7030-0c86-4612-9238-4a3e1b0075b7" (UID: "706e7030-0c86-4612-9238-4a3e1b0075b7"). InnerVolumeSpecName "kube-api-access-9bcld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:25.026304 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:25.026273 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/706e7030-0c86-4612-9238-4a3e1b0075b7-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:25.026304 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:25.026301 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bcld\" (UniqueName: \"kubernetes.io/projected/706e7030-0c86-4612-9238-4a3e1b0075b7-kube-api-access-9bcld\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:25.691768 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:25.691734 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj" Oct 29 23:31:25.691768 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:25.691755 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj" event={"ID":"706e7030-0c86-4612-9238-4a3e1b0075b7","Type":"ContainerDied","Data":"ffefec53f652c3f1a7b84aa4b5335da409a329e89d26c17eeb7ae1133d841446"} Oct 29 23:31:25.691960 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:25.691802 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffefec53f652c3f1a7b84aa4b5335da409a329e89d26c17eeb7ae1133d841446" Oct 29 23:31:26.582134 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:26.582095 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w"] Oct 29 23:31:26.689217 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:26.689178 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:31:26.770238 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:26.770188 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963cedd3-0342-431f-939b-627bd09b94c4" path="/var/lib/kubelet/pods/963cedd3-0342-431f-939b-627bd09b94c4/volumes" Oct 29 23:31:31.047518 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:31.047222 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk"] Oct 29 23:31:31.052488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:31.052186 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e9924e-fkfqk"] Oct 29 23:31:32.169129 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:32.169093 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps"] Oct 29 23:31:32.169613 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:32.169382 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" podUID="6bf4f192-0296-4a17-abf7-b6a55b12f60c" containerName="istio-proxy" containerID="cri-o://c36f2552dece2633e15fc2be06bb4b920a1aad6d558614c333798c005faaff69" gracePeriod=30 Oct 29 23:31:32.175545 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:32.175511 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd"] Oct 29 23:31:32.175829 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:32.175797 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" podUID="991d97b5-f7ae-4895-a247-79b1200eec8b" containerName="istio-proxy" containerID="cri-o://2371d91b0f42e9f41f5950256024ad0a2eda950abe01831c2315dd48bdbf923f" gracePeriod=30 Oct 29 23:31:32.181757 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:32.181721 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86"] Oct 29 23:31:32.181956 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:32.181935 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" podUID="77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" containerName="istio-proxy" containerID="cri-o://6397bdc304d6a63a78bdcd971dc5c03b09f522a189a93e851b1e92c7e0c9813a" gracePeriod=30 Oct 29 23:31:32.362746 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:32.362694 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg"] Oct 29 23:31:32.395197 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:32.395158 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp"] Oct 29 23:31:32.610014 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:32.609977 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:31:32.610196 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:32.610038 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" podUID="991d97b5-f7ae-4895-a247-79b1200eec8b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:31:32.782440 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:32.782398 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921" path="/var/lib/kubelet/pods/55b9a5ae-c4b9-4eff-b421-9f6c2d2b7921/volumes" Oct 29 23:31:33.209386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.209342 6507 patch_prober.go:29] interesting pod/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:31:33.209834 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.209415 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" podUID="77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:31:33.778594 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.778559 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h"] Oct 29 23:31:33.779050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.779036 6507 cpu_manager.go:414] "containerMap: removing container" podUID="706e7030-0c86-4612-9238-4a3e1b0075b7" containerName="revoke-keys" Oct 29 23:31:33.779104 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.779051 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706e7030-0c86-4612-9238-4a3e1b0075b7" containerName="revoke-keys" Oct 29 23:31:33.779327 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.779126 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="706e7030-0c86-4612-9238-4a3e1b0075b7" containerName="revoke-keys" Oct 29 23:31:33.783519 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.783495 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" Oct 29 23:31:33.802585 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.802549 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h"] Oct 29 23:31:33.808113 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.808086 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:31:33.808369 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.808351 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:31:33.808469 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.808442 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:31:33.808537 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.808526 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:31:33.838734 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.838690 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65kgw\" (UniqueName: \"kubernetes.io/projected/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-kube-api-access-65kgw\") pod \"maas-api-revoke-sub-fc0f98f9-hxv9h\" (UID: \"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" Oct 29 23:31:33.838944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.838846 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc0f98f9-hxv9h\" (UID: \"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" Oct 29 23:31:33.839027 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.838999 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc0f98f9-hxv9h\" (UID: \"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" Oct 29 23:31:33.940657 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.940613 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65kgw\" (UniqueName: \"kubernetes.io/projected/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-kube-api-access-65kgw\") pod \"maas-api-revoke-sub-fc0f98f9-hxv9h\" (UID: \"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" Oct 29 23:31:33.940657 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.940661 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc0f98f9-hxv9h\" (UID: \"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" Oct 29 23:31:33.940922 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.940719 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc0f98f9-hxv9h\" (UID: \"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" Oct 29 23:31:33.941353 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.941330 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc0f98f9-hxv9h\" (UID: \"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" Oct 29 23:31:33.943068 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.943048 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc0f98f9-hxv9h\" (UID: \"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" Oct 29 23:31:33.957476 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:33.957435 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65kgw\" (UniqueName: \"kubernetes.io/projected/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-kube-api-access-65kgw\") pod \"maas-api-revoke-sub-fc0f98f9-hxv9h\" (UID: \"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" Oct 29 23:31:34.027063 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:34.027029 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x"] Oct 29 23:31:34.030969 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:34.030899 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3325f452-zxc9x"] Oct 29 23:31:34.098866 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:34.098821 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" Oct 29 23:31:34.401123 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:34.401094 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h"] Oct 29 23:31:34.403569 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:31:34.403539 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d07b2cd_119b_4fdb_a067_7adeec5d5b2c.slice/crio-e0e0d63924d88dab8b987884f57b99526d46200f22f96e5d62ece18974627725 WatchSource:0}: Error finding container e0e0d63924d88dab8b987884f57b99526d46200f22f96e5d62ece18974627725: Status 404 returned error can't find the container with id e0e0d63924d88dab8b987884f57b99526d46200f22f96e5d62ece18974627725 Oct 29 23:31:34.753335 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:34.753298 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8" path="/var/lib/kubelet/pods/6ffe3bc1-f6ca-4ffd-a481-f3fcf5ee18b8/volumes" Oct 29 23:31:34.806086 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:34.806045 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" event={"ID":"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c","Type":"ContainerStarted","Data":"468d95d1846e38b7841ff06d0804cd411e5c7a4f77ea26df2734233e77775c9c"} Oct 29 23:31:34.806254 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:34.806106 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" event={"ID":"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c","Type":"ContainerStarted","Data":"e0e0d63924d88dab8b987884f57b99526d46200f22f96e5d62ece18974627725"} Oct 29 23:31:34.843764 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:34.843713 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" podStartSLOduration=1.843693954 podStartE2EDuration="1.843693954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:34.842137288 +0000 UTC m=+1064.205142013" watchObservedRunningTime="2026-10-29 23:31:34.843693954 +0000 UTC m=+1064.206698674" Oct 29 23:31:34.866236 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:34.866202 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:31:34.968379 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:34.968299 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp"] Oct 29 23:31:34.972517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:34.972493 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp" Oct 29 23:31:34.991658 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:34.991628 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp"] Oct 29 23:31:35.026767 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:35.026722 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds"] Oct 29 23:31:35.030612 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:35.030572 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0df51531-6wvds"] Oct 29 23:31:35.051237 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:35.051200 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d8948a-c28f-4e73-9f44-776139987374-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp\" (UID: \"f4d8948a-c28f-4e73-9f44-776139987374\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp" Oct 29 23:31:35.051411 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:35.051324 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqgm8\" (UniqueName: \"kubernetes.io/projected/f4d8948a-c28f-4e73-9f44-776139987374-kube-api-access-pqgm8\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp\" (UID: \"f4d8948a-c28f-4e73-9f44-776139987374\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp" Oct 29 23:31:35.051488 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:35.051434 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d8948a-c28f-4e73-9f44-776139987374-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp\" (UID: \"f4d8948a-c28f-4e73-9f44-776139987374\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp" Oct 29 23:31:35.152633 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:35.152585 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d8948a-c28f-4e73-9f44-776139987374-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp\" (UID: \"f4d8948a-c28f-4e73-9f44-776139987374\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp" Oct 29 23:31:35.152831 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:35.152700 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d8948a-c28f-4e73-9f44-776139987374-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp\" (UID: \"f4d8948a-c28f-4e73-9f44-776139987374\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp" Oct 29 23:31:35.152831 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:35.152804 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqgm8\" (UniqueName: \"kubernetes.io/projected/f4d8948a-c28f-4e73-9f44-776139987374-kube-api-access-pqgm8\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp\" (UID: \"f4d8948a-c28f-4e73-9f44-776139987374\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp" Oct 29 23:31:35.153405 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:35.153380 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d8948a-c28f-4e73-9f44-776139987374-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp\" (UID: \"f4d8948a-c28f-4e73-9f44-776139987374\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp" Oct 29 23:31:35.155105 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:35.155078 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d8948a-c28f-4e73-9f44-776139987374-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp\" (UID: \"f4d8948a-c28f-4e73-9f44-776139987374\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp" Oct 29 23:31:35.172894 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:35.172859 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqgm8\" (UniqueName: \"kubernetes.io/projected/f4d8948a-c28f-4e73-9f44-776139987374-kube-api-access-pqgm8\") pod \"maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp\" (UID: \"f4d8948a-c28f-4e73-9f44-776139987374\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp" Oct 29 23:31:35.286918 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:35.286873 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp" Oct 29 23:31:35.642770 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:35.642733 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp"] Oct 29 23:31:35.645397 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:31:35.645368 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d8948a_c28f_4e73_9f44_776139987374.slice/crio-680af78cb7d827484700aa896f8e3edabee3b9ed75e346edf9a0cb3cbd9f8b76 WatchSource:0}: Error finding container 680af78cb7d827484700aa896f8e3edabee3b9ed75e346edf9a0cb3cbd9f8b76: Status 404 returned error can't find the container with id 680af78cb7d827484700aa896f8e3edabee3b9ed75e346edf9a0cb3cbd9f8b76 Oct 29 23:31:35.813863 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:35.813824 6507 generic.go:356] "Generic (PLEG): container finished" podID="9d07b2cd-119b-4fdb-a067-7adeec5d5b2c" containerID="468d95d1846e38b7841ff06d0804cd411e5c7a4f77ea26df2734233e77775c9c" exitCode=0 Oct 29 23:31:35.814053 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:35.813900 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" event={"ID":"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c","Type":"ContainerDied","Data":"468d95d1846e38b7841ff06d0804cd411e5c7a4f77ea26df2734233e77775c9c"} Oct 29 23:31:35.815316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:35.815288 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp" event={"ID":"f4d8948a-c28f-4e73-9f44-776139987374","Type":"ContainerStarted","Data":"680af78cb7d827484700aa896f8e3edabee3b9ed75e346edf9a0cb3cbd9f8b76"} Oct 29 23:31:36.753995 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:36.753958 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eef75b8-42b2-42b7-8950-f62cc548cea0" path="/var/lib/kubelet/pods/7eef75b8-42b2-42b7-8950-f62cc548cea0/volumes" Oct 29 23:31:36.823895 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:36.823809 6507 generic.go:356] "Generic (PLEG): container finished" podID="f4d8948a-c28f-4e73-9f44-776139987374" containerID="e81cc6c99ad32489a654cfad34f098acf62fef155da8044fdfb617a7e45b0e7f" exitCode=0 Oct 29 23:31:36.824049 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:36.823894 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp" event={"ID":"f4d8948a-c28f-4e73-9f44-776139987374","Type":"ContainerDied","Data":"e81cc6c99ad32489a654cfad34f098acf62fef155da8044fdfb617a7e45b0e7f"} Oct 29 23:31:36.999768 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:36.999734 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" Oct 29 23:31:37.102189 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.102097 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-maas-api-xks-ca\") pod \"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c\" (UID: \"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c\") " Oct 29 23:31:37.102189 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.102149 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-kube-api-access-65kgw\" (UniqueName: \"kubernetes.io/projected/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-kube-api-access-65kgw\") pod \"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c\" (UID: \"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c\") " Oct 29 23:31:37.102189 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.102171 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-maas-api-openshift-ca\") pod \"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c\" (UID: \"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c\") " Oct 29 23:31:37.102674 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.102648 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-maas-api-openshift-ca" pod "9d07b2cd-119b-4fdb-a067-7adeec5d5b2c" (UID: "9d07b2cd-119b-4fdb-a067-7adeec5d5b2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:37.104354 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.104325 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-maas-api-xks-ca" pod "9d07b2cd-119b-4fdb-a067-7adeec5d5b2c" (UID: "9d07b2cd-119b-4fdb-a067-7adeec5d5b2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:37.104496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.104385 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-kube-api-access-65kgw" pod "9d07b2cd-119b-4fdb-a067-7adeec5d5b2c" (UID: "9d07b2cd-119b-4fdb-a067-7adeec5d5b2c"). InnerVolumeSpecName "kube-api-access-65kgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:37.203209 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.203176 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.203209 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.203207 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65kgw\" (UniqueName: \"kubernetes.io/projected/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-kube-api-access-65kgw\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.203376 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.203219 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.363541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.363467 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" podUID="80680fb1-ec38-4c40-8b92-ca49981bf54a" containerName="payload-processing" containerID="cri-o://860f1424f34b78fe28cfcccb0a16b18c226f77cabfa17d718e773fc48fc4780d" gracePeriod=25 Oct 29 23:31:37.396347 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.396293 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" podUID="77a9bc98-cb47-403d-b1ea-824dde7ae578" containerName="payload-pre-processing" containerID="cri-o://e30269bd21b2a956143814f669d70a262597d660daf9e25a09925e1985df8888" gracePeriod=25 Oct 29 23:31:37.443450 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.443413 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:31:37.472768 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.472732 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:31:37.510680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.510645 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:31:37.608305 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608222 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-workload-socket\") pod \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " Oct 29 23:31:37.608305 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608267 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-envoy\") pod \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " Oct 29 23:31:37.608305 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608303 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-envoy\") pod \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " Oct 29 23:31:37.608611 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608339 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/991d97b5-f7ae-4895-a247-79b1200eec8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991d97b5-f7ae-4895-a247-79b1200eec8b-istiod-ca-cert\") pod \"991d97b5-f7ae-4895-a247-79b1200eec8b\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " Oct 29 23:31:37.608611 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608373 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-envoy\") pod \"991d97b5-f7ae-4895-a247-79b1200eec8b\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " Oct 29 23:31:37.608611 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608406 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-credential-socket\") pod \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " Oct 29 23:31:37.608611 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608440 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-token\" (UniqueName: \"kubernetes.io/projected/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-token\") pod \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " Oct 29 23:31:37.608611 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608491 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-workload-certs\") pod \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " Oct 29 23:31:37.608611 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608545 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-data\") pod \"991d97b5-f7ae-4895-a247-79b1200eec8b\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " Oct 29 23:31:37.608611 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608580 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/991d97b5-f7ae-4895-a247-79b1200eec8b-kube-api-access-j687m\" (UniqueName: \"kubernetes.io/projected/991d97b5-f7ae-4895-a247-79b1200eec8b-kube-api-access-j687m\") pod \"991d97b5-f7ae-4895-a247-79b1200eec8b\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " Oct 29 23:31:37.609037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608641 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-workload-socket\") pod \"991d97b5-f7ae-4895-a247-79b1200eec8b\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " Oct 29 23:31:37.609037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608659 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-workload-socket" pod "6bf4f192-0296-4a17-abf7-b6a55b12f60c" (UID: "6bf4f192-0296-4a17-abf7-b6a55b12f60c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:37.609037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608689 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istiod-ca-cert\") pod \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " Oct 29 23:31:37.609037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608733 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-podinfo\") pod \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " Oct 29 23:31:37.609037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608757 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991d97b5-f7ae-4895-a247-79b1200eec8b-istiod-ca-cert" pod "991d97b5-f7ae-4895-a247-79b1200eec8b" (UID: "991d97b5-f7ae-4895-a247-79b1200eec8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:37.609037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608774 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-workload-certs\") pod \"991d97b5-f7ae-4895-a247-79b1200eec8b\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " Oct 29 23:31:37.609037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608828 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-credential-socket\") pod \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " Oct 29 23:31:37.609037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608854 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-token\" (UniqueName: \"kubernetes.io/projected/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-token\") pod \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " Oct 29 23:31:37.609037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608885 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf4f192-0296-4a17-abf7-b6a55b12f60c-kube-api-access-qjh9q\" (UniqueName: \"kubernetes.io/projected/6bf4f192-0296-4a17-abf7-b6a55b12f60c-kube-api-access-qjh9q\") pod \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " Oct 29 23:31:37.609037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608964 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-workload-certs\") pod \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " Oct 29 23:31:37.609037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.609005 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-podinfo\") pod \"991d97b5-f7ae-4895-a247-79b1200eec8b\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " Oct 29 23:31:37.609709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.609073 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-data\") pod \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " Oct 29 23:31:37.609709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.609125 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-token\" (UniqueName: \"kubernetes.io/projected/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-token\") pod \"991d97b5-f7ae-4895-a247-79b1200eec8b\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " Oct 29 23:31:37.609709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.609162 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-credential-socket\") pod \"991d97b5-f7ae-4895-a247-79b1200eec8b\" (UID: \"991d97b5-f7ae-4895-a247-79b1200eec8b\") " Oct 29 23:31:37.609709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.609194 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-data\") pod \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " Oct 29 23:31:37.609709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.609235 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istiod-ca-cert\") pod \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " Oct 29 23:31:37.609709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.608884 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-data" pod "991d97b5-f7ae-4895-a247-79b1200eec8b" (UID: "991d97b5-f7ae-4895-a247-79b1200eec8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:37.609709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.609246 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istiod-ca-cert" pod "6bf4f192-0296-4a17-abf7-b6a55b12f60c" (UID: "6bf4f192-0296-4a17-abf7-b6a55b12f60c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:37.609709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.609115 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-workload-certs" pod "6bf4f192-0296-4a17-abf7-b6a55b12f60c" (UID: "6bf4f192-0296-4a17-abf7-b6a55b12f60c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:37.609709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.609225 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-credential-socket" pod "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" (UID: "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:37.609709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.609287 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-podinfo\") pod \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\" (UID: \"6bf4f192-0296-4a17-abf7-b6a55b12f60c\") " Oct 29 23:31:37.609709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.609329 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-kube-api-access-jqnv8\" (UniqueName: \"kubernetes.io/projected/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-kube-api-access-jqnv8\") pod \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " Oct 29 23:31:37.609709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.609368 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-workload-socket\") pod \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\" (UID: \"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50\") " Oct 29 23:31:37.609709 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.609523 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-workload-socket" pod "991d97b5-f7ae-4895-a247-79b1200eec8b" (UID: "991d97b5-f7ae-4895-a247-79b1200eec8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:37.610502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.609721 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-credential-socket" pod "6bf4f192-0296-4a17-abf7-b6a55b12f60c" (UID: "6bf4f192-0296-4a17-abf7-b6a55b12f60c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:37.610502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.609842 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-credential-socket" pod "991d97b5-f7ae-4895-a247-79b1200eec8b" (UID: "991d97b5-f7ae-4895-a247-79b1200eec8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:37.610502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.609913 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-workload-certs" pod "991d97b5-f7ae-4895-a247-79b1200eec8b" (UID: "991d97b5-f7ae-4895-a247-79b1200eec8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:37.610502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.610034 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-data" pod "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" (UID: "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:37.610502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.610096 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.610502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.610117 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.610502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.610132 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991d97b5-f7ae-4895-a247-79b1200eec8b-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.610502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.610146 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.610502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.610158 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.610502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.610163 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-data" pod "6bf4f192-0296-4a17-abf7-b6a55b12f60c" (UID: "6bf4f192-0296-4a17-abf7-b6a55b12f60c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:37.610502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.610172 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.610502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.610156 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-workload-certs" pod "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" (UID: "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:37.610502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.610185 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.610502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.610198 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.610502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.610213 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.610502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.610227 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.611430 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.610582 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istiod-ca-cert" pod "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" (UID: "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:37.611430 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.610900 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-workload-socket" pod "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" (UID: "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:37.611924 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.611865 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-envoy" pod "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" (UID: "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:37.612061 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.612038 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-envoy" pod "6bf4f192-0296-4a17-abf7-b6a55b12f60c" (UID: "6bf4f192-0296-4a17-abf7-b6a55b12f60c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:37.613403 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.613357 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-envoy" pod "991d97b5-f7ae-4895-a247-79b1200eec8b" (UID: "991d97b5-f7ae-4895-a247-79b1200eec8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:37.613811 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.613780 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-podinfo" pod "991d97b5-f7ae-4895-a247-79b1200eec8b" (UID: "991d97b5-f7ae-4895-a247-79b1200eec8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:31:37.614102 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.614030 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991d97b5-f7ae-4895-a247-79b1200eec8b-kube-api-access-j687m" pod "991d97b5-f7ae-4895-a247-79b1200eec8b" (UID: "991d97b5-f7ae-4895-a247-79b1200eec8b"). InnerVolumeSpecName "kube-api-access-j687m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:37.614348 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.614204 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-token" pod "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" (UID: "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:37.614587 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.614560 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-podinfo" pod "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" (UID: "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:31:37.614816 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.614784 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf4f192-0296-4a17-abf7-b6a55b12f60c-kube-api-access-qjh9q" pod "6bf4f192-0296-4a17-abf7-b6a55b12f60c" (UID: "6bf4f192-0296-4a17-abf7-b6a55b12f60c"). InnerVolumeSpecName "kube-api-access-qjh9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:37.615209 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.615184 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-kube-api-access-jqnv8" pod "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" (UID: "77d1d7e4-9a92-4ffa-9cb8-20dee7904a50"). InnerVolumeSpecName "kube-api-access-jqnv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:37.615638 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.615591 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-podinfo" pod "6bf4f192-0296-4a17-abf7-b6a55b12f60c" (UID: "6bf4f192-0296-4a17-abf7-b6a55b12f60c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:31:37.615723 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.615673 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-token" pod "991d97b5-f7ae-4895-a247-79b1200eec8b" (UID: "991d97b5-f7ae-4895-a247-79b1200eec8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:37.616366 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.616331 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-token" pod "6bf4f192-0296-4a17-abf7-b6a55b12f60c" (UID: "6bf4f192-0296-4a17-abf7-b6a55b12f60c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:37.711441 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711383 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.711646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711446 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j687m\" (UniqueName: \"kubernetes.io/projected/991d97b5-f7ae-4895-a247-79b1200eec8b-kube-api-access-j687m\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.711646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711501 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.711646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711515 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.711646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711529 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjh9q\" (UniqueName: \"kubernetes.io/projected/6bf4f192-0296-4a17-abf7-b6a55b12f60c-kube-api-access-qjh9q\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.711646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711543 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.711646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711557 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.711646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711572 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.711646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711587 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.711646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711598 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.711646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711610 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.711646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711622 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.711646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711635 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqnv8\" (UniqueName: \"kubernetes.io/projected/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-kube-api-access-jqnv8\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.711646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711648 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.712069 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711664 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bf4f192-0296-4a17-abf7-b6a55b12f60c-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.712069 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711678 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.712069 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.711691 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991d97b5-f7ae-4895-a247-79b1200eec8b-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:37.831506 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.831451 6507 generic.go:356] "Generic (PLEG): container finished" podID="77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" containerID="6397bdc304d6a63a78bdcd971dc5c03b09f522a189a93e851b1e92c7e0c9813a" exitCode=0 Oct 29 23:31:37.831506 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.831484 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" event={"ID":"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50","Type":"ContainerDied","Data":"6397bdc304d6a63a78bdcd971dc5c03b09f522a189a93e851b1e92c7e0c9813a"} Oct 29 23:31:37.832043 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.831527 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" Oct 29 23:31:37.832043 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.831539 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86" event={"ID":"77d1d7e4-9a92-4ffa-9cb8-20dee7904a50","Type":"ContainerDied","Data":"0f666c9ed0f4a2ce7663173fba4c9ba6c03f6c057d3b9d5a7e9ca4e13480fccf"} Oct 29 23:31:37.832043 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.831562 6507 scope.go:122] "RemoveContainer" containerID="6397bdc304d6a63a78bdcd971dc5c03b09f522a189a93e851b1e92c7e0c9813a" Oct 29 23:31:37.833007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.832983 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" Oct 29 23:31:37.833121 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.833014 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h" event={"ID":"9d07b2cd-119b-4fdb-a067-7adeec5d5b2c","Type":"ContainerDied","Data":"e0e0d63924d88dab8b987884f57b99526d46200f22f96e5d62ece18974627725"} Oct 29 23:31:37.833121 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.833059 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0e0d63924d88dab8b987884f57b99526d46200f22f96e5d62ece18974627725" Oct 29 23:31:37.834533 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.834505 6507 generic.go:356] "Generic (PLEG): container finished" podID="991d97b5-f7ae-4895-a247-79b1200eec8b" containerID="2371d91b0f42e9f41f5950256024ad0a2eda950abe01831c2315dd48bdbf923f" exitCode=0 Oct 29 23:31:37.834634 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.834554 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" Oct 29 23:31:37.834634 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.834573 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" event={"ID":"991d97b5-f7ae-4895-a247-79b1200eec8b","Type":"ContainerDied","Data":"2371d91b0f42e9f41f5950256024ad0a2eda950abe01831c2315dd48bdbf923f"} Oct 29 23:31:37.834634 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.834628 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd" event={"ID":"991d97b5-f7ae-4895-a247-79b1200eec8b","Type":"ContainerDied","Data":"f65e7ba29d77b27092f530e71c8839e2f29d65ef1007c016dccf7d7736045744"} Oct 29 23:31:37.840262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.840237 6507 generic.go:356] "Generic (PLEG): container finished" podID="6bf4f192-0296-4a17-abf7-b6a55b12f60c" containerID="c36f2552dece2633e15fc2be06bb4b920a1aad6d558614c333798c005faaff69" exitCode=0 Oct 29 23:31:37.840522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.840447 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" event={"ID":"6bf4f192-0296-4a17-abf7-b6a55b12f60c","Type":"ContainerDied","Data":"c36f2552dece2633e15fc2be06bb4b920a1aad6d558614c333798c005faaff69"} Oct 29 23:31:37.840522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.840504 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" Oct 29 23:31:37.840702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.840506 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps" event={"ID":"6bf4f192-0296-4a17-abf7-b6a55b12f60c","Type":"ContainerDied","Data":"ce047f8142524409485c03919e25e1007452abd0ae47417f0f9f6c8a3aac7969"} Oct 29 23:31:37.845622 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.845601 6507 scope.go:122] "RemoveContainer" containerID="6397bdc304d6a63a78bdcd971dc5c03b09f522a189a93e851b1e92c7e0c9813a" Oct 29 23:31:37.845938 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:31:37.845918 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6397bdc304d6a63a78bdcd971dc5c03b09f522a189a93e851b1e92c7e0c9813a\": container with ID starting with 6397bdc304d6a63a78bdcd971dc5c03b09f522a189a93e851b1e92c7e0c9813a not found: ID does not exist" containerID="6397bdc304d6a63a78bdcd971dc5c03b09f522a189a93e851b1e92c7e0c9813a" Oct 29 23:31:37.846018 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.845946 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6397bdc304d6a63a78bdcd971dc5c03b09f522a189a93e851b1e92c7e0c9813a"} err="failed to get container status \"6397bdc304d6a63a78bdcd971dc5c03b09f522a189a93e851b1e92c7e0c9813a\": rpc error: code = NotFound desc = could not find container \"6397bdc304d6a63a78bdcd971dc5c03b09f522a189a93e851b1e92c7e0c9813a\": container with ID starting with 6397bdc304d6a63a78bdcd971dc5c03b09f522a189a93e851b1e92c7e0c9813a not found: ID does not exist" Oct 29 23:31:37.846018 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.845961 6507 scope.go:122] "RemoveContainer" containerID="2371d91b0f42e9f41f5950256024ad0a2eda950abe01831c2315dd48bdbf923f" Oct 29 23:31:37.857710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.857686 6507 scope.go:122] "RemoveContainer" containerID="2371d91b0f42e9f41f5950256024ad0a2eda950abe01831c2315dd48bdbf923f" Oct 29 23:31:37.858035 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:31:37.858014 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2371d91b0f42e9f41f5950256024ad0a2eda950abe01831c2315dd48bdbf923f\": container with ID starting with 2371d91b0f42e9f41f5950256024ad0a2eda950abe01831c2315dd48bdbf923f not found: ID does not exist" containerID="2371d91b0f42e9f41f5950256024ad0a2eda950abe01831c2315dd48bdbf923f" Oct 29 23:31:37.858102 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.858044 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2371d91b0f42e9f41f5950256024ad0a2eda950abe01831c2315dd48bdbf923f"} err="failed to get container status \"2371d91b0f42e9f41f5950256024ad0a2eda950abe01831c2315dd48bdbf923f\": rpc error: code = NotFound desc = could not find container \"2371d91b0f42e9f41f5950256024ad0a2eda950abe01831c2315dd48bdbf923f\": container with ID starting with 2371d91b0f42e9f41f5950256024ad0a2eda950abe01831c2315dd48bdbf923f not found: ID does not exist" Oct 29 23:31:37.858102 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.858064 6507 scope.go:122] "RemoveContainer" containerID="c36f2552dece2633e15fc2be06bb4b920a1aad6d558614c333798c005faaff69" Oct 29 23:31:37.868685 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.868659 6507 scope.go:122] "RemoveContainer" containerID="21ac6a31e98cbcf718f5c10e510b40b8fda554c41f516d4b6842a0a56a583170" Oct 29 23:31:37.881156 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.881122 6507 scope.go:122] "RemoveContainer" containerID="c36f2552dece2633e15fc2be06bb4b920a1aad6d558614c333798c005faaff69" Oct 29 23:31:37.881691 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:31:37.881632 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36f2552dece2633e15fc2be06bb4b920a1aad6d558614c333798c005faaff69\": container with ID starting with c36f2552dece2633e15fc2be06bb4b920a1aad6d558614c333798c005faaff69 not found: ID does not exist" containerID="c36f2552dece2633e15fc2be06bb4b920a1aad6d558614c333798c005faaff69" Oct 29 23:31:37.881789 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.881700 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36f2552dece2633e15fc2be06bb4b920a1aad6d558614c333798c005faaff69"} err="failed to get container status \"c36f2552dece2633e15fc2be06bb4b920a1aad6d558614c333798c005faaff69\": rpc error: code = NotFound desc = could not find container \"c36f2552dece2633e15fc2be06bb4b920a1aad6d558614c333798c005faaff69\": container with ID starting with c36f2552dece2633e15fc2be06bb4b920a1aad6d558614c333798c005faaff69 not found: ID does not exist" Oct 29 23:31:37.881789 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.881728 6507 scope.go:122] "RemoveContainer" containerID="21ac6a31e98cbcf718f5c10e510b40b8fda554c41f516d4b6842a0a56a583170" Oct 29 23:31:37.882096 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:31:37.882040 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ac6a31e98cbcf718f5c10e510b40b8fda554c41f516d4b6842a0a56a583170\": container with ID starting with 21ac6a31e98cbcf718f5c10e510b40b8fda554c41f516d4b6842a0a56a583170 not found: ID does not exist" containerID="21ac6a31e98cbcf718f5c10e510b40b8fda554c41f516d4b6842a0a56a583170" Oct 29 23:31:37.882096 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.882067 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ac6a31e98cbcf718f5c10e510b40b8fda554c41f516d4b6842a0a56a583170"} err="failed to get container status \"21ac6a31e98cbcf718f5c10e510b40b8fda554c41f516d4b6842a0a56a583170\": rpc error: code = NotFound desc = could not find container \"21ac6a31e98cbcf718f5c10e510b40b8fda554c41f516d4b6842a0a56a583170\": container with ID starting with 21ac6a31e98cbcf718f5c10e510b40b8fda554c41f516d4b6842a0a56a583170 not found: ID does not exist" Oct 29 23:31:37.905392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.905298 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86"] Oct 29 23:31:37.909001 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.908969 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-cxc86"] Oct 29 23:31:37.946764 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.946726 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd"] Oct 29 23:31:37.952828 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.952793 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-mxhsd"] Oct 29 23:31:37.968932 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.968902 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp" Oct 29 23:31:37.983822 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.983791 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps"] Oct 29 23:31:37.987531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:37.987505 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-be8726-openshift-default-55dcd5768f-bj7ps"] Oct 29 23:31:38.115694 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:38.115608 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4d8948a-c28f-4e73-9f44-776139987374-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d8948a-c28f-4e73-9f44-776139987374-maas-api-xks-ca\") pod \"f4d8948a-c28f-4e73-9f44-776139987374\" (UID: \"f4d8948a-c28f-4e73-9f44-776139987374\") " Oct 29 23:31:38.115694 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:38.115671 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d8948a-c28f-4e73-9f44-776139987374-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d8948a-c28f-4e73-9f44-776139987374-maas-api-openshift-ca\") pod \"f4d8948a-c28f-4e73-9f44-776139987374\" (UID: \"f4d8948a-c28f-4e73-9f44-776139987374\") " Oct 29 23:31:38.115928 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:38.115758 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d8948a-c28f-4e73-9f44-776139987374-kube-api-access-pqgm8\" (UniqueName: \"kubernetes.io/projected/f4d8948a-c28f-4e73-9f44-776139987374-kube-api-access-pqgm8\") pod \"f4d8948a-c28f-4e73-9f44-776139987374\" (UID: \"f4d8948a-c28f-4e73-9f44-776139987374\") " Oct 29 23:31:38.116080 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:38.116051 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d8948a-c28f-4e73-9f44-776139987374-maas-api-openshift-ca" pod "f4d8948a-c28f-4e73-9f44-776139987374" (UID: "f4d8948a-c28f-4e73-9f44-776139987374"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:38.116221 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:38.116179 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d8948a-c28f-4e73-9f44-776139987374-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:38.117885 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:38.117860 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4d8948a-c28f-4e73-9f44-776139987374-maas-api-xks-ca" pod "f4d8948a-c28f-4e73-9f44-776139987374" (UID: "f4d8948a-c28f-4e73-9f44-776139987374"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:38.117996 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:38.117976 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d8948a-c28f-4e73-9f44-776139987374-kube-api-access-pqgm8" pod "f4d8948a-c28f-4e73-9f44-776139987374" (UID: "f4d8948a-c28f-4e73-9f44-776139987374"). InnerVolumeSpecName "kube-api-access-pqgm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:38.217115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:38.217069 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d8948a-c28f-4e73-9f44-776139987374-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:38.217115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:38.217108 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqgm8\" (UniqueName: \"kubernetes.io/projected/f4d8948a-c28f-4e73-9f44-776139987374-kube-api-access-pqgm8\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:38.752777 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:38.752736 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bf4f192-0296-4a17-abf7-b6a55b12f60c" path="/var/lib/kubelet/pods/6bf4f192-0296-4a17-abf7-b6a55b12f60c/volumes" Oct 29 23:31:38.753207 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:38.753193 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" path="/var/lib/kubelet/pods/77d1d7e4-9a92-4ffa-9cb8-20dee7904a50/volumes" Oct 29 23:31:38.753616 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:38.753603 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="991d97b5-f7ae-4895-a247-79b1200eec8b" path="/var/lib/kubelet/pods/991d97b5-f7ae-4895-a247-79b1200eec8b/volumes" Oct 29 23:31:38.853072 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:38.853041 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp" Oct 29 23:31:38.853072 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:38.853057 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp" event={"ID":"f4d8948a-c28f-4e73-9f44-776139987374","Type":"ContainerDied","Data":"680af78cb7d827484700aa896f8e3edabee3b9ed75e346edf9a0cb3cbd9f8b76"} Oct 29 23:31:38.853542 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:38.853105 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="680af78cb7d827484700aa896f8e3edabee3b9ed75e346edf9a0cb3cbd9f8b76" Oct 29 23:31:39.030419 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:39.030330 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs"] Oct 29 23:31:39.033793 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:39.033757 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dac35cd-c7bcs"] Oct 29 23:31:40.497541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:40.497497 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" podUID="77a9bc98-cb47-403d-b1ea-824dde7ae578" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:40.497961 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:40.497509 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" podUID="80680fb1-ec38-4c40-8b92-ca49981bf54a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:40.752399 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:40.752312 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0baf759f-868e-47bc-b515-ae615233f4bf" path="/var/lib/kubelet/pods/0baf759f-868e-47bc-b515-ae615233f4bf/volumes" Oct 29 23:31:41.603616 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.603569 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" containerID="cri-o://70645e011d51298d95c5be17b01181d0ef0e4d450bc1698c827661f0a3e93e6e" gracePeriod=45 Oct 29 23:31:41.868476 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.868372 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:31:41.874877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.874838 6507 generic.go:356] "Generic (PLEG): container finished" podID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerID="70645e011d51298d95c5be17b01181d0ef0e4d450bc1698c827661f0a3e93e6e" exitCode=0 Oct 29 23:31:41.875045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.874885 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" Oct 29 23:31:41.875045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.874919 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" event={"ID":"f783e5d3-00f5-4110-af04-0dc0cabf30f5","Type":"ContainerDied","Data":"70645e011d51298d95c5be17b01181d0ef0e4d450bc1698c827661f0a3e93e6e"} Oct 29 23:31:41.875045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.874968 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w" event={"ID":"f783e5d3-00f5-4110-af04-0dc0cabf30f5","Type":"ContainerDied","Data":"1685f8c773692bb59b8627889fed626f474697e7ad5a3a98e0c28928ce709265"} Oct 29 23:31:41.875045 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.874988 6507 scope.go:122] "RemoveContainer" containerID="70645e011d51298d95c5be17b01181d0ef0e4d450bc1698c827661f0a3e93e6e" Oct 29 23:31:41.887043 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.887006 6507 scope.go:122] "RemoveContainer" containerID="70645e011d51298d95c5be17b01181d0ef0e4d450bc1698c827661f0a3e93e6e" Oct 29 23:31:41.887432 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:31:41.887408 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70645e011d51298d95c5be17b01181d0ef0e4d450bc1698c827661f0a3e93e6e\": container with ID starting with 70645e011d51298d95c5be17b01181d0ef0e4d450bc1698c827661f0a3e93e6e not found: ID does not exist" containerID="70645e011d51298d95c5be17b01181d0ef0e4d450bc1698c827661f0a3e93e6e" Oct 29 23:31:41.887572 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.887440 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70645e011d51298d95c5be17b01181d0ef0e4d450bc1698c827661f0a3e93e6e"} err="failed to get container status \"70645e011d51298d95c5be17b01181d0ef0e4d450bc1698c827661f0a3e93e6e\": rpc error: code = NotFound desc = could not find container \"70645e011d51298d95c5be17b01181d0ef0e4d450bc1698c827661f0a3e93e6e\": container with ID starting with 70645e011d51298d95c5be17b01181d0ef0e4d450bc1698c827661f0a3e93e6e not found: ID does not exist" Oct 29 23:31:41.958014 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.957969 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-dshm\") pod \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " Oct 29 23:31:41.958228 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.958123 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-tmp-dir\") pod \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " Oct 29 23:31:41.958297 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.958239 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f783e5d3-00f5-4110-af04-0dc0cabf30f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/f783e5d3-00f5-4110-af04-0dc0cabf30f5-tls-certs\") pod \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " Oct 29 23:31:41.958297 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.958270 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-model-cache\") pod \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " Oct 29 23:31:41.958407 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.958334 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f783e5d3-00f5-4110-af04-0dc0cabf30f5-kube-api-access-fb49g\" (UniqueName: \"kubernetes.io/projected/f783e5d3-00f5-4110-af04-0dc0cabf30f5-kube-api-access-fb49g\") pod \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " Oct 29 23:31:41.958407 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.958369 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-home\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-home\") pod \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\" (UID: \"f783e5d3-00f5-4110-af04-0dc0cabf30f5\") " Oct 29 23:31:41.958573 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.958329 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-tmp-dir" pod "f783e5d3-00f5-4110-af04-0dc0cabf30f5" (UID: "f783e5d3-00f5-4110-af04-0dc0cabf30f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:41.958573 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.958557 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-model-cache" pod "f783e5d3-00f5-4110-af04-0dc0cabf30f5" (UID: "f783e5d3-00f5-4110-af04-0dc0cabf30f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:41.958861 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.958806 6507 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-tmp-dir\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:41.958861 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.958836 6507 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-model-cache\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:41.958861 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.958806 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-home" pod "f783e5d3-00f5-4110-af04-0dc0cabf30f5" (UID: "f783e5d3-00f5-4110-af04-0dc0cabf30f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:41.961136 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.961105 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f783e5d3-00f5-4110-af04-0dc0cabf30f5-kube-api-access-fb49g" pod "f783e5d3-00f5-4110-af04-0dc0cabf30f5" (UID: "f783e5d3-00f5-4110-af04-0dc0cabf30f5"). InnerVolumeSpecName "kube-api-access-fb49g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:41.961229 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.961108 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-dshm" pod "f783e5d3-00f5-4110-af04-0dc0cabf30f5" (UID: "f783e5d3-00f5-4110-af04-0dc0cabf30f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:41.961313 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:41.961297 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f783e5d3-00f5-4110-af04-0dc0cabf30f5-tls-certs" pod "f783e5d3-00f5-4110-af04-0dc0cabf30f5" (UID: "f783e5d3-00f5-4110-af04-0dc0cabf30f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:42.059790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:42.059757 6507 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f783e5d3-00f5-4110-af04-0dc0cabf30f5-tls-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:42.059790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:42.059789 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb49g\" (UniqueName: \"kubernetes.io/projected/f783e5d3-00f5-4110-af04-0dc0cabf30f5-kube-api-access-fb49g\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:42.060004 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:42.059801 6507 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-home\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:42.060004 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:42.059814 6507 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f783e5d3-00f5-4110-af04-0dc0cabf30f5-dshm\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:42.235392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:42.235357 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w"] Oct 29 23:31:42.240584 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:42.240550 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9280e4/premium-w3-9280e4-kserve-6d9bc9fbfc-dpg2w"] Oct 29 23:31:42.752356 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:42.752319 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" path="/var/lib/kubelet/pods/f783e5d3-00f5-4110-af04-0dc0cabf30f5/volumes" Oct 29 23:31:43.030847 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:43.030745 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm"] Oct 29 23:31:43.039910 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:43.039867 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8401733a-mhbvm"] Oct 29 23:31:44.752477 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:44.752432 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16329f0e-c15c-4841-946d-c5df1de04a6b" path="/var/lib/kubelet/pods/16329f0e-c15c-4841-946d-c5df1de04a6b/volumes" Oct 29 23:31:45.224745 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:31:45.224704 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:31:45.490418 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:45.490328 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn"] Oct 29 23:31:45.524308 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:45.524267 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st"] Oct 29 23:31:45.905189 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:45.905107 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:31:46.930650 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:46.930611 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj"] Oct 29 23:31:46.939576 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:46.939533 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9280e4-s2zhj"] Oct 29 23:31:48.300423 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.300385 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn"] Oct 29 23:31:48.300874 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.300855 6507 cpu_manager.go:414] "containerMap: removing container" podUID="77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" containerName="istio-proxy" Oct 29 23:31:48.300874 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.300868 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" containerName="istio-proxy" Oct 29 23:31:48.300957 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.300885 6507 cpu_manager.go:414] "containerMap: removing container" podUID="6bf4f192-0296-4a17-abf7-b6a55b12f60c" containerName="istio-proxy" Oct 29 23:31:48.300957 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.300891 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf4f192-0296-4a17-abf7-b6a55b12f60c" containerName="istio-proxy" Oct 29 23:31:48.300957 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.300901 6507 cpu_manager.go:414] "containerMap: removing container" podUID="9d07b2cd-119b-4fdb-a067-7adeec5d5b2c" containerName="revoke-keys" Oct 29 23:31:48.300957 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.300906 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d07b2cd-119b-4fdb-a067-7adeec5d5b2c" containerName="revoke-keys" Oct 29 23:31:48.300957 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.300915 6507 cpu_manager.go:414] "containerMap: removing container" podUID="f4d8948a-c28f-4e73-9f44-776139987374" containerName="revoke-keys" Oct 29 23:31:48.300957 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.300920 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d8948a-c28f-4e73-9f44-776139987374" containerName="revoke-keys" Oct 29 23:31:48.300957 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.300931 6507 cpu_manager.go:414] "containerMap: removing container" podUID="991d97b5-f7ae-4895-a247-79b1200eec8b" containerName="istio-proxy" Oct 29 23:31:48.300957 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.300935 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991d97b5-f7ae-4895-a247-79b1200eec8b" containerName="istio-proxy" Oct 29 23:31:48.300957 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.300956 6507 cpu_manager.go:414] "containerMap: removing container" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" Oct 29 23:31:48.301252 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.300961 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" Oct 29 23:31:48.301252 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.301025 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="f783e5d3-00f5-4110-af04-0dc0cabf30f5" containerName="main" Oct 29 23:31:48.301252 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.301034 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf4f192-0296-4a17-abf7-b6a55b12f60c" containerName="istio-proxy" Oct 29 23:31:48.301252 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.301041 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d1d7e4-9a92-4ffa-9cb8-20dee7904a50" containerName="istio-proxy" Oct 29 23:31:48.301252 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.301047 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf4f192-0296-4a17-abf7-b6a55b12f60c" containerName="istio-proxy" Oct 29 23:31:48.301252 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.301054 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d07b2cd-119b-4fdb-a067-7adeec5d5b2c" containerName="revoke-keys" Oct 29 23:31:48.301252 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.301060 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d8948a-c28f-4e73-9f44-776139987374" containerName="revoke-keys" Oct 29 23:31:48.301252 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.301066 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="991d97b5-f7ae-4895-a247-79b1200eec8b" containerName="istio-proxy" Oct 29 23:31:48.305744 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.305728 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn" Oct 29 23:31:48.319442 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.319413 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:31:48.329036 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.329003 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:31:48.329036 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.329040 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:31:48.329276 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.329011 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:31:48.332591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.332563 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn"] Oct 29 23:31:48.353180 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.353143 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m728r\" (UniqueName: \"kubernetes.io/projected/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-kube-api-access-m728r\") pod \"maas-api-revoke-sub-7b0df1f2-qlvvn\" (UID: \"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn" Oct 29 23:31:48.353399 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.353372 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b0df1f2-qlvvn\" (UID: \"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn" Oct 29 23:31:48.353475 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.353423 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b0df1f2-qlvvn\" (UID: \"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn" Oct 29 23:31:48.454279 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.454246 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b0df1f2-qlvvn\" (UID: \"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn" Oct 29 23:31:48.454279 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.454283 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b0df1f2-qlvvn\" (UID: \"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn" Oct 29 23:31:48.454533 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.454326 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m728r\" (UniqueName: \"kubernetes.io/projected/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-kube-api-access-m728r\") pod \"maas-api-revoke-sub-7b0df1f2-qlvvn\" (UID: \"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn" Oct 29 23:31:48.454953 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.454921 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b0df1f2-qlvvn\" (UID: \"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn" Oct 29 23:31:48.456750 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.456727 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b0df1f2-qlvvn\" (UID: \"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn" Oct 29 23:31:48.470696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.470668 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m728r\" (UniqueName: \"kubernetes.io/projected/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-kube-api-access-m728r\") pod \"maas-api-revoke-sub-7b0df1f2-qlvvn\" (UID: \"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn" Oct 29 23:31:48.619649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.619565 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn" Oct 29 23:31:48.752334 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.752303 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="706e7030-0c86-4612-9238-4a3e1b0075b7" path="/var/lib/kubelet/pods/706e7030-0c86-4612-9238-4a3e1b0075b7/volumes" Oct 29 23:31:48.917126 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.916971 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn"] Oct 29 23:31:48.920172 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:31:48.920127 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a425f6_b9c8_40a4_a403_fbd3a6d7baa4.slice/crio-2499b8c160bc2f59a23de72345f8d11b26d1a95e7d9bc23929cd5a756eaab358 WatchSource:0}: Error finding container 2499b8c160bc2f59a23de72345f8d11b26d1a95e7d9bc23929cd5a756eaab358: Status 404 returned error can't find the container with id 2499b8c160bc2f59a23de72345f8d11b26d1a95e7d9bc23929cd5a756eaab358 Oct 29 23:31:48.951107 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:48.951074 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn" event={"ID":"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4","Type":"ContainerStarted","Data":"2499b8c160bc2f59a23de72345f8d11b26d1a95e7d9bc23929cd5a756eaab358"} Oct 29 23:31:49.026421 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:49.026382 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks"] Oct 29 23:31:49.029214 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:49.029182 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c73a3f-l9zks"] Oct 29 23:31:49.958558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:49.958523 6507 generic.go:356] "Generic (PLEG): container finished" podID="22a425f6-b9c8-40a4-a403-fbd3a6d7baa4" containerID="9a96951b8483bcbf963b853b9bc292d022ce379f8fe82c0ffcc879704330ff8e" exitCode=0 Oct 29 23:31:49.958963 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:49.958612 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn" event={"ID":"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4","Type":"ContainerDied","Data":"9a96951b8483bcbf963b853b9bc292d022ce379f8fe82c0ffcc879704330ff8e"} Oct 29 23:31:50.026389 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:50.026302 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf"] Oct 29 23:31:50.030632 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:50.030604 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54ef1cd5-65nmf"] Oct 29 23:31:50.491024 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:50.490978 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" podUID="55ba2c05-8056-4ccc-beeb-118dff6c1875" containerName="payload-processing" containerID="cri-o://cb339fcf1cdfae9cb8c567d1841f8cb599fefedb8ee21fb04a65b53a850408eb" gracePeriod=25 Oct 29 23:31:50.496635 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:50.496603 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" podUID="77a9bc98-cb47-403d-b1ea-824dde7ae578" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:50.496757 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:50.496603 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" podUID="80680fb1-ec38-4c40-8b92-ca49981bf54a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:50.525235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:50.525193 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" podUID="ee03dd91-8ebd-4268-9fbc-9017f73223b0" containerName="payload-pre-processing" containerID="cri-o://2a0e1f64eddc36d0540327e09c054b18c06a7007ab4dfd76302b7e9035d1d90e" gracePeriod=25 Oct 29 23:31:50.753279 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:50.753206 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208de026-0c8d-4e9f-a6dc-6fdcc394c4ca" path="/var/lib/kubelet/pods/208de026-0c8d-4e9f-a6dc-6fdcc394c4ca/volumes" Oct 29 23:31:50.753639 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:50.753625 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7803083-cb6b-4c2c-91a0-8ed401b2e794" path="/var/lib/kubelet/pods/c7803083-cb6b-4c2c-91a0-8ed401b2e794/volumes" Oct 29 23:31:51.100416 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:51.100393 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn" Oct 29 23:31:51.181870 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:51.181840 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-maas-api-openshift-ca\") pod \"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4\" (UID: \"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4\") " Oct 29 23:31:51.182036 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:51.181928 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-kube-api-access-m728r\" (UniqueName: \"kubernetes.io/projected/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-kube-api-access-m728r\") pod \"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4\" (UID: \"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4\") " Oct 29 23:31:51.182128 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:51.182110 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-maas-api-xks-ca\") pod \"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4\" (UID: \"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4\") " Oct 29 23:31:51.182188 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:51.182160 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-maas-api-openshift-ca" pod "22a425f6-b9c8-40a4-a403-fbd3a6d7baa4" (UID: "22a425f6-b9c8-40a4-a403-fbd3a6d7baa4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:51.182447 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:51.182428 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:51.184126 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:51.184103 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-maas-api-xks-ca" pod "22a425f6-b9c8-40a4-a403-fbd3a6d7baa4" (UID: "22a425f6-b9c8-40a4-a403-fbd3a6d7baa4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:51.184126 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:51.184117 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-kube-api-access-m728r" pod "22a425f6-b9c8-40a4-a403-fbd3a6d7baa4" (UID: "22a425f6-b9c8-40a4-a403-fbd3a6d7baa4"). InnerVolumeSpecName "kube-api-access-m728r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:51.290292 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:51.290203 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m728r\" (UniqueName: \"kubernetes.io/projected/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-kube-api-access-m728r\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:51.294547 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:51.294514 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:51.973255 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:51.973227 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn" Oct 29 23:31:51.973442 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:51.973262 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn" event={"ID":"22a425f6-b9c8-40a4-a403-fbd3a6d7baa4","Type":"ContainerDied","Data":"2499b8c160bc2f59a23de72345f8d11b26d1a95e7d9bc23929cd5a756eaab358"} Oct 29 23:31:51.973442 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:51.973309 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2499b8c160bc2f59a23de72345f8d11b26d1a95e7d9bc23929cd5a756eaab358" Oct 29 23:31:52.567612 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:52.567584 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" podUID="ee03dd91-8ebd-4268-9fbc-9017f73223b0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:53.571397 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:53.571361 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" podUID="55ba2c05-8056-4ccc-beeb-118dff6c1875" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:31:53.989690 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:53.989659 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h"] Oct 29 23:31:53.989910 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:53.989887 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" podUID="80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" containerName="istio-proxy" containerID="cri-o://578a760b59cea6f1cc063bc47b2a5086b5228b24a3499e4db6363f60410c59bf" gracePeriod=30 Oct 29 23:31:53.998796 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:53.998767 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc"] Oct 29 23:31:53.998973 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:53.998953 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" podUID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerName="istio-proxy" containerID="cri-o://cc5383e2bac9805fb9dd29aca5128fa5132932aad0f3e6ec04ee74428f8a6acb" gracePeriod=30 Oct 29 23:31:54.131311 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:54.131273 6507 patch_prober.go:29] interesting pod/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:31:54.131514 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:54.131345 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" podUID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:31:55.560871 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:55.560824 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:31:57.027446 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:57.027414 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2"] Oct 29 23:31:57.036629 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:57.036597 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18d78c27-h5pt2"] Oct 29 23:31:57.651265 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:57.651237 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" Oct 29 23:31:57.680964 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:57.680940 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" Oct 29 23:31:57.766360 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:57.766320 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80680fb1-ec38-4c40-8b92-ca49981bf54a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80680fb1-ec38-4c40-8b92-ca49981bf54a-plugins-config-volume\") pod \"80680fb1-ec38-4c40-8b92-ca49981bf54a\" (UID: \"80680fb1-ec38-4c40-8b92-ca49981bf54a\") " Oct 29 23:31:57.766571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:57.766474 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77a9bc98-cb47-403d-b1ea-824dde7ae578-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77a9bc98-cb47-403d-b1ea-824dde7ae578-plugins-config-volume\") pod \"77a9bc98-cb47-403d-b1ea-824dde7ae578\" (UID: \"77a9bc98-cb47-403d-b1ea-824dde7ae578\") " Oct 29 23:31:57.766571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:57.766529 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a9bc98-cb47-403d-b1ea-824dde7ae578-kube-api-access-54qnf\" (UniqueName: \"kubernetes.io/projected/77a9bc98-cb47-403d-b1ea-824dde7ae578-kube-api-access-54qnf\") pod \"77a9bc98-cb47-403d-b1ea-824dde7ae578\" (UID: \"77a9bc98-cb47-403d-b1ea-824dde7ae578\") " Oct 29 23:31:57.766687 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:57.766641 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80680fb1-ec38-4c40-8b92-ca49981bf54a-kube-api-access-pq5kg\" (UniqueName: \"kubernetes.io/projected/80680fb1-ec38-4c40-8b92-ca49981bf54a-kube-api-access-pq5kg\") pod \"80680fb1-ec38-4c40-8b92-ca49981bf54a\" (UID: \"80680fb1-ec38-4c40-8b92-ca49981bf54a\") " Oct 29 23:31:57.766855 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:57.766826 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80680fb1-ec38-4c40-8b92-ca49981bf54a-plugins-config-volume" pod "80680fb1-ec38-4c40-8b92-ca49981bf54a" (UID: "80680fb1-ec38-4c40-8b92-ca49981bf54a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:57.766855 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:57.766843 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a9bc98-cb47-403d-b1ea-824dde7ae578-plugins-config-volume" pod "77a9bc98-cb47-403d-b1ea-824dde7ae578" (UID: "77a9bc98-cb47-403d-b1ea-824dde7ae578"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:57.768797 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:57.767800 6507 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77a9bc98-cb47-403d-b1ea-824dde7ae578-plugins-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:57.768797 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:57.767844 6507 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80680fb1-ec38-4c40-8b92-ca49981bf54a-plugins-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:57.775842 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:57.775809 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a9bc98-cb47-403d-b1ea-824dde7ae578-kube-api-access-54qnf" pod "77a9bc98-cb47-403d-b1ea-824dde7ae578" (UID: "77a9bc98-cb47-403d-b1ea-824dde7ae578"). InnerVolumeSpecName "kube-api-access-54qnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:57.775842 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:57.775819 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80680fb1-ec38-4c40-8b92-ca49981bf54a-kube-api-access-pq5kg" pod "80680fb1-ec38-4c40-8b92-ca49981bf54a" (UID: "80680fb1-ec38-4c40-8b92-ca49981bf54a"). InnerVolumeSpecName "kube-api-access-pq5kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:57.868746 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:57.868708 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54qnf\" (UniqueName: \"kubernetes.io/projected/77a9bc98-cb47-403d-b1ea-824dde7ae578-kube-api-access-54qnf\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:57.868746 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:57.868742 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq5kg\" (UniqueName: \"kubernetes.io/projected/80680fb1-ec38-4c40-8b92-ca49981bf54a-kube-api-access-pq5kg\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:58.018922 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.018885 6507 generic.go:356] "Generic (PLEG): container finished" podID="77a9bc98-cb47-403d-b1ea-824dde7ae578" containerID="e30269bd21b2a956143814f669d70a262597d660daf9e25a09925e1985df8888" exitCode=0 Oct 29 23:31:58.019103 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.018959 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" Oct 29 23:31:58.019103 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.018972 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" event={"ID":"77a9bc98-cb47-403d-b1ea-824dde7ae578","Type":"ContainerDied","Data":"e30269bd21b2a956143814f669d70a262597d660daf9e25a09925e1985df8888"} Oct 29 23:31:58.019103 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.019071 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp" event={"ID":"77a9bc98-cb47-403d-b1ea-824dde7ae578","Type":"ContainerDied","Data":"c5730d40d9fb90da35ea6ef5640621ad4fa2dde49ee72596d143d49baaaed5d6"} Oct 29 23:31:58.019287 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.019101 6507 scope.go:122] "RemoveContainer" containerID="e30269bd21b2a956143814f669d70a262597d660daf9e25a09925e1985df8888" Oct 29 23:31:58.020414 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.020391 6507 generic.go:356] "Generic (PLEG): container finished" podID="80680fb1-ec38-4c40-8b92-ca49981bf54a" containerID="860f1424f34b78fe28cfcccb0a16b18c226f77cabfa17d718e773fc48fc4780d" exitCode=0 Oct 29 23:31:58.020548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.020447 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" Oct 29 23:31:58.020548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.020502 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" event={"ID":"80680fb1-ec38-4c40-8b92-ca49981bf54a","Type":"ContainerDied","Data":"860f1424f34b78fe28cfcccb0a16b18c226f77cabfa17d718e773fc48fc4780d"} Oct 29 23:31:58.020548 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.020544 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg" event={"ID":"80680fb1-ec38-4c40-8b92-ca49981bf54a","Type":"ContainerDied","Data":"f1a81f84d1845ceb50867eddf9a6b7811ce31ff84bb850ba4bbcefbbea6715c3"} Oct 29 23:31:58.028979 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.028954 6507 scope.go:122] "RemoveContainer" containerID="e30269bd21b2a956143814f669d70a262597d660daf9e25a09925e1985df8888" Oct 29 23:31:58.029319 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:31:58.029274 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30269bd21b2a956143814f669d70a262597d660daf9e25a09925e1985df8888\": container with ID starting with e30269bd21b2a956143814f669d70a262597d660daf9e25a09925e1985df8888 not found: ID does not exist" containerID="e30269bd21b2a956143814f669d70a262597d660daf9e25a09925e1985df8888" Oct 29 23:31:58.029319 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.029302 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30269bd21b2a956143814f669d70a262597d660daf9e25a09925e1985df8888"} err="failed to get container status \"e30269bd21b2a956143814f669d70a262597d660daf9e25a09925e1985df8888\": rpc error: code = NotFound desc = could not find container \"e30269bd21b2a956143814f669d70a262597d660daf9e25a09925e1985df8888\": container with ID starting with e30269bd21b2a956143814f669d70a262597d660daf9e25a09925e1985df8888 not found: ID does not exist" Oct 29 23:31:58.029418 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.029321 6507 scope.go:122] "RemoveContainer" containerID="860f1424f34b78fe28cfcccb0a16b18c226f77cabfa17d718e773fc48fc4780d" Oct 29 23:31:58.038924 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.038904 6507 scope.go:122] "RemoveContainer" containerID="860f1424f34b78fe28cfcccb0a16b18c226f77cabfa17d718e773fc48fc4780d" Oct 29 23:31:58.039182 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:31:58.039165 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"860f1424f34b78fe28cfcccb0a16b18c226f77cabfa17d718e773fc48fc4780d\": container with ID starting with 860f1424f34b78fe28cfcccb0a16b18c226f77cabfa17d718e773fc48fc4780d not found: ID does not exist" containerID="860f1424f34b78fe28cfcccb0a16b18c226f77cabfa17d718e773fc48fc4780d" Oct 29 23:31:58.039227 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.039190 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"860f1424f34b78fe28cfcccb0a16b18c226f77cabfa17d718e773fc48fc4780d"} err="failed to get container status \"860f1424f34b78fe28cfcccb0a16b18c226f77cabfa17d718e773fc48fc4780d\": rpc error: code = NotFound desc = could not find container \"860f1424f34b78fe28cfcccb0a16b18c226f77cabfa17d718e773fc48fc4780d\": container with ID starting with 860f1424f34b78fe28cfcccb0a16b18c226f77cabfa17d718e773fc48fc4780d not found: ID does not exist" Oct 29 23:31:58.061453 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.061424 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg"] Oct 29 23:31:58.066800 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.066778 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9280e4-68cb5c6bf6-t4qtg"] Oct 29 23:31:58.094054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.094005 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp"] Oct 29 23:31:58.099934 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.099903 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9280e4-7d9b499975-744gp"] Oct 29 23:31:58.752601 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.752567 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="423fb383-714f-478d-b772-0d7678789327" path="/var/lib/kubelet/pods/423fb383-714f-478d-b772-0d7678789327/volumes" Oct 29 23:31:58.752968 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.752955 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77a9bc98-cb47-403d-b1ea-824dde7ae578" path="/var/lib/kubelet/pods/77a9bc98-cb47-403d-b1ea-824dde7ae578/volumes" Oct 29 23:31:58.753279 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:58.753267 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80680fb1-ec38-4c40-8b92-ca49981bf54a" path="/var/lib/kubelet/pods/80680fb1-ec38-4c40-8b92-ca49981bf54a/volumes" Oct 29 23:31:59.261078 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.260981 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:31:59.276495 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.276447 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:31:59.383566 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.383536 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-data\") pod \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " Oct 29 23:31:59.383715 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.383593 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-podinfo\") pod \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " Oct 29 23:31:59.383715 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.383625 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-kube-api-access-7wltr\" (UniqueName: \"kubernetes.io/projected/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-kube-api-access-7wltr\") pod \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " Oct 29 23:31:59.383715 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.383663 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-envoy\") pod \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " Oct 29 23:31:59.383715 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.383700 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-podinfo\") pod \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " Oct 29 23:31:59.383948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.383729 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-credential-socket\") pod \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " Oct 29 23:31:59.383948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.383771 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-data\") pod \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " Oct 29 23:31:59.383948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.383803 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-kube-api-access-hfjmk\" (UniqueName: \"kubernetes.io/projected/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-kube-api-access-hfjmk\") pod \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " Oct 29 23:31:59.383948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.383808 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-data" pod "dac751e7-bb4e-4bf7-81f2-26fee5a478f3" (UID: "dac751e7-bb4e-4bf7-81f2-26fee5a478f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:59.383948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.383833 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-workload-certs\") pod \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " Oct 29 23:31:59.383948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.383882 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-workload-socket\") pod \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " Oct 29 23:31:59.384262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.383958 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-token\" (UniqueName: \"kubernetes.io/projected/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-token\") pod \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " Oct 29 23:31:59.384262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.383997 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-envoy\") pod \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " Oct 29 23:31:59.384262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.384035 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-workload-certs\") pod \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " Oct 29 23:31:59.384262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.384099 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istiod-ca-cert\") pod \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " Oct 29 23:31:59.384262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.384143 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-token\" (UniqueName: \"kubernetes.io/projected/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-token\") pod \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " Oct 29 23:31:59.384262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.384183 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-credential-socket\") pod \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " Oct 29 23:31:59.384262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.384207 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istiod-ca-cert\") pod \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\" (UID: \"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c\") " Oct 29 23:31:59.384262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.384260 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-workload-socket\") pod \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\" (UID: \"dac751e7-bb4e-4bf7-81f2-26fee5a478f3\") " Oct 29 23:31:59.384726 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.384651 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.384726 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.384685 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-workload-certs" pod "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" (UID: "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:59.385132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.384792 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-credential-socket" pod "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" (UID: "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:59.385132 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.384887 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-workload-socket" pod "dac751e7-bb4e-4bf7-81f2-26fee5a478f3" (UID: "dac751e7-bb4e-4bf7-81f2-26fee5a478f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:59.385316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.385034 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-data" pod "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" (UID: "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:59.385316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.385060 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-workload-socket" pod "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" (UID: "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:59.385506 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.385449 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istiod-ca-cert" pod "dac751e7-bb4e-4bf7-81f2-26fee5a478f3" (UID: "dac751e7-bb4e-4bf7-81f2-26fee5a478f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:59.385949 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.385921 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-credential-socket" pod "dac751e7-bb4e-4bf7-81f2-26fee5a478f3" (UID: "dac751e7-bb4e-4bf7-81f2-26fee5a478f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:59.386421 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.386392 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-podinfo" pod "dac751e7-bb4e-4bf7-81f2-26fee5a478f3" (UID: "dac751e7-bb4e-4bf7-81f2-26fee5a478f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:31:59.386763 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.386738 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istiod-ca-cert" pod "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" (UID: "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:59.387027 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.387005 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-workload-certs" pod "dac751e7-bb4e-4bf7-81f2-26fee5a478f3" (UID: "dac751e7-bb4e-4bf7-81f2-26fee5a478f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:59.387391 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.387366 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-kube-api-access-7wltr" pod "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" (UID: "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c"). InnerVolumeSpecName "kube-api-access-7wltr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:59.388302 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.388278 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-envoy" pod "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" (UID: "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:59.389486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.389437 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-podinfo" pod "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" (UID: "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:31:59.389600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.389534 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-token" pod "dac751e7-bb4e-4bf7-81f2-26fee5a478f3" (UID: "dac751e7-bb4e-4bf7-81f2-26fee5a478f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:59.389666 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.389596 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-kube-api-access-hfjmk" pod "dac751e7-bb4e-4bf7-81f2-26fee5a478f3" (UID: "dac751e7-bb4e-4bf7-81f2-26fee5a478f3"). InnerVolumeSpecName "kube-api-access-hfjmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:59.389728 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.389709 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-envoy" pod "dac751e7-bb4e-4bf7-81f2-26fee5a478f3" (UID: "dac751e7-bb4e-4bf7-81f2-26fee5a478f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:59.389909 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.389888 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-token" pod "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" (UID: "80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:59.485747 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485710 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.485747 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485741 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.485747 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485752 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.485987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485760 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.485987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485769 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.485987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485776 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.485987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485784 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.485987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485792 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.485987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485801 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.485987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485810 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wltr\" (UniqueName: \"kubernetes.io/projected/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-kube-api-access-7wltr\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.485987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485818 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.485987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485829 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.485987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485837 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.485987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485846 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.485987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485853 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfjmk\" (UniqueName: \"kubernetes.io/projected/dac751e7-bb4e-4bf7-81f2-26fee5a478f3-kube-api-access-hfjmk\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.485987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485861 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:31:59.485987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:31:59.485869 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:00.038101 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.038064 6507 generic.go:356] "Generic (PLEG): container finished" podID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerID="cc5383e2bac9805fb9dd29aca5128fa5132932aad0f3e6ec04ee74428f8a6acb" exitCode=0 Oct 29 23:32:00.038248 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.038126 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" Oct 29 23:32:00.038248 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.038137 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" event={"ID":"dac751e7-bb4e-4bf7-81f2-26fee5a478f3","Type":"ContainerDied","Data":"cc5383e2bac9805fb9dd29aca5128fa5132932aad0f3e6ec04ee74428f8a6acb"} Oct 29 23:32:00.038248 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.038198 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc" event={"ID":"dac751e7-bb4e-4bf7-81f2-26fee5a478f3","Type":"ContainerDied","Data":"dd0292da219233ffc9fc1a6feae2e2a09d6f678bcc5e77fce45d2ed98db9e305"} Oct 29 23:32:00.038248 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.038242 6507 scope.go:122] "RemoveContainer" containerID="cc5383e2bac9805fb9dd29aca5128fa5132932aad0f3e6ec04ee74428f8a6acb" Oct 29 23:32:00.039679 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.039656 6507 generic.go:356] "Generic (PLEG): container finished" podID="80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" containerID="578a760b59cea6f1cc063bc47b2a5086b5228b24a3499e4db6363f60410c59bf" exitCode=0 Oct 29 23:32:00.039784 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.039686 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" event={"ID":"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c","Type":"ContainerDied","Data":"578a760b59cea6f1cc063bc47b2a5086b5228b24a3499e4db6363f60410c59bf"} Oct 29 23:32:00.039784 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.039721 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" Oct 29 23:32:00.039784 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.039731 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h" event={"ID":"80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c","Type":"ContainerDied","Data":"79b0b8940aa021e53ebac5b8ac3d365a9e40b67a3e4682632aa27e494a5115bc"} Oct 29 23:32:00.048058 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.048039 6507 scope.go:122] "RemoveContainer" containerID="606ae1be70bbb339138254abb97a9ce15e8334b24e3344ff19e42a6c40ff9567" Oct 29 23:32:00.057056 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.057040 6507 scope.go:122] "RemoveContainer" containerID="cc5383e2bac9805fb9dd29aca5128fa5132932aad0f3e6ec04ee74428f8a6acb" Oct 29 23:32:00.057305 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:32:00.057284 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc5383e2bac9805fb9dd29aca5128fa5132932aad0f3e6ec04ee74428f8a6acb\": container with ID starting with cc5383e2bac9805fb9dd29aca5128fa5132932aad0f3e6ec04ee74428f8a6acb not found: ID does not exist" containerID="cc5383e2bac9805fb9dd29aca5128fa5132932aad0f3e6ec04ee74428f8a6acb" Oct 29 23:32:00.057354 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.057311 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc5383e2bac9805fb9dd29aca5128fa5132932aad0f3e6ec04ee74428f8a6acb"} err="failed to get container status \"cc5383e2bac9805fb9dd29aca5128fa5132932aad0f3e6ec04ee74428f8a6acb\": rpc error: code = NotFound desc = could not find container \"cc5383e2bac9805fb9dd29aca5128fa5132932aad0f3e6ec04ee74428f8a6acb\": container with ID starting with cc5383e2bac9805fb9dd29aca5128fa5132932aad0f3e6ec04ee74428f8a6acb not found: ID does not exist" Oct 29 23:32:00.057354 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.057336 6507 scope.go:122] "RemoveContainer" containerID="606ae1be70bbb339138254abb97a9ce15e8334b24e3344ff19e42a6c40ff9567" Oct 29 23:32:00.057573 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:32:00.057557 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606ae1be70bbb339138254abb97a9ce15e8334b24e3344ff19e42a6c40ff9567\": container with ID starting with 606ae1be70bbb339138254abb97a9ce15e8334b24e3344ff19e42a6c40ff9567 not found: ID does not exist" containerID="606ae1be70bbb339138254abb97a9ce15e8334b24e3344ff19e42a6c40ff9567" Oct 29 23:32:00.057622 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.057579 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606ae1be70bbb339138254abb97a9ce15e8334b24e3344ff19e42a6c40ff9567"} err="failed to get container status \"606ae1be70bbb339138254abb97a9ce15e8334b24e3344ff19e42a6c40ff9567\": rpc error: code = NotFound desc = could not find container \"606ae1be70bbb339138254abb97a9ce15e8334b24e3344ff19e42a6c40ff9567\": container with ID starting with 606ae1be70bbb339138254abb97a9ce15e8334b24e3344ff19e42a6c40ff9567 not found: ID does not exist" Oct 29 23:32:00.057622 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.057595 6507 scope.go:122] "RemoveContainer" containerID="578a760b59cea6f1cc063bc47b2a5086b5228b24a3499e4db6363f60410c59bf" Oct 29 23:32:00.067610 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.067593 6507 scope.go:122] "RemoveContainer" containerID="578a760b59cea6f1cc063bc47b2a5086b5228b24a3499e4db6363f60410c59bf" Oct 29 23:32:00.067870 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:32:00.067851 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578a760b59cea6f1cc063bc47b2a5086b5228b24a3499e4db6363f60410c59bf\": container with ID starting with 578a760b59cea6f1cc063bc47b2a5086b5228b24a3499e4db6363f60410c59bf not found: ID does not exist" containerID="578a760b59cea6f1cc063bc47b2a5086b5228b24a3499e4db6363f60410c59bf" Oct 29 23:32:00.067912 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.067877 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578a760b59cea6f1cc063bc47b2a5086b5228b24a3499e4db6363f60410c59bf"} err="failed to get container status \"578a760b59cea6f1cc063bc47b2a5086b5228b24a3499e4db6363f60410c59bf\": rpc error: code = NotFound desc = could not find container \"578a760b59cea6f1cc063bc47b2a5086b5228b24a3499e4db6363f60410c59bf\": container with ID starting with 578a760b59cea6f1cc063bc47b2a5086b5228b24a3499e4db6363f60410c59bf not found: ID does not exist" Oct 29 23:32:00.092796 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.092775 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h"] Oct 29 23:32:00.101981 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.101960 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-87x4h"] Oct 29 23:32:00.126830 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.126802 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc"] Oct 29 23:32:00.130268 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.130247 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9280e4-openshift-default-6c769b66b7-v9pvc"] Oct 29 23:32:00.752129 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.752051 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" path="/var/lib/kubelet/pods/80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c/volumes" Oct 29 23:32:00.752551 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:00.752537 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" path="/var/lib/kubelet/pods/dac751e7-bb4e-4bf7-81f2-26fee5a478f3/volumes" Oct 29 23:32:02.026244 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:02.026213 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j"] Oct 29 23:32:02.035815 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:02.035788 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e5a66bd-9f78j"] Oct 29 23:32:02.566716 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:02.566674 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" podUID="ee03dd91-8ebd-4268-9fbc-9017f73223b0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:32:02.753021 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:02.752974 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07008149-7cf0-44aa-a624-c2bd45af32f8" path="/var/lib/kubelet/pods/07008149-7cf0-44aa-a624-c2bd45af32f8/volumes" Oct 29 23:32:03.113652 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:03.113612 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:32:03.571676 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:03.571635 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" podUID="55ba2c05-8056-4ccc-beeb-118dff6c1875" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:32:03.999964 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:03.999851 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp"] Oct 29 23:32:04.009259 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:04.009219 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0b2ae9-z4lpp"] Oct 29 23:32:04.752517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:04.752479 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d8948a-c28f-4e73-9f44-776139987374" path="/var/lib/kubelet/pods/f4d8948a-c28f-4e73-9f44-776139987374/volumes" Oct 29 23:32:09.772071 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:09.772037 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:32:10.781978 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:10.781953 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" Oct 29 23:32:10.825691 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:10.825645 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" Oct 29 23:32:10.918875 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:10.918843 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55ba2c05-8056-4ccc-beeb-118dff6c1875-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55ba2c05-8056-4ccc-beeb-118dff6c1875-plugins-config-volume\") pod \"55ba2c05-8056-4ccc-beeb-118dff6c1875\" (UID: \"55ba2c05-8056-4ccc-beeb-118dff6c1875\") " Oct 29 23:32:10.918875 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:10.918886 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee03dd91-8ebd-4268-9fbc-9017f73223b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee03dd91-8ebd-4268-9fbc-9017f73223b0-plugins-config-volume\") pod \"ee03dd91-8ebd-4268-9fbc-9017f73223b0\" (UID: \"ee03dd91-8ebd-4268-9fbc-9017f73223b0\") " Oct 29 23:32:10.919105 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:10.919006 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ba2c05-8056-4ccc-beeb-118dff6c1875-kube-api-access-74xj6\" (UniqueName: \"kubernetes.io/projected/55ba2c05-8056-4ccc-beeb-118dff6c1875-kube-api-access-74xj6\") pod \"55ba2c05-8056-4ccc-beeb-118dff6c1875\" (UID: \"55ba2c05-8056-4ccc-beeb-118dff6c1875\") " Oct 29 23:32:10.919105 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:10.919028 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee03dd91-8ebd-4268-9fbc-9017f73223b0-kube-api-access-5kd27\" (UniqueName: \"kubernetes.io/projected/ee03dd91-8ebd-4268-9fbc-9017f73223b0-kube-api-access-5kd27\") pod \"ee03dd91-8ebd-4268-9fbc-9017f73223b0\" (UID: \"ee03dd91-8ebd-4268-9fbc-9017f73223b0\") " Oct 29 23:32:10.919318 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:10.919283 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ba2c05-8056-4ccc-beeb-118dff6c1875-plugins-config-volume" pod "55ba2c05-8056-4ccc-beeb-118dff6c1875" (UID: "55ba2c05-8056-4ccc-beeb-118dff6c1875"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:10.919318 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:10.919287 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee03dd91-8ebd-4268-9fbc-9017f73223b0-plugins-config-volume" pod "ee03dd91-8ebd-4268-9fbc-9017f73223b0" (UID: "ee03dd91-8ebd-4268-9fbc-9017f73223b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:10.921102 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:10.921075 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee03dd91-8ebd-4268-9fbc-9017f73223b0-kube-api-access-5kd27" pod "ee03dd91-8ebd-4268-9fbc-9017f73223b0" (UID: "ee03dd91-8ebd-4268-9fbc-9017f73223b0"). InnerVolumeSpecName "kube-api-access-5kd27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:10.921198 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:10.921179 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ba2c05-8056-4ccc-beeb-118dff6c1875-kube-api-access-74xj6" pod "55ba2c05-8056-4ccc-beeb-118dff6c1875" (UID: "55ba2c05-8056-4ccc-beeb-118dff6c1875"). InnerVolumeSpecName "kube-api-access-74xj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:11.020816 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.020735 6507 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee03dd91-8ebd-4268-9fbc-9017f73223b0-plugins-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:11.020816 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.020762 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74xj6\" (UniqueName: \"kubernetes.io/projected/55ba2c05-8056-4ccc-beeb-118dff6c1875-kube-api-access-74xj6\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:11.020816 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.020770 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kd27\" (UniqueName: \"kubernetes.io/projected/ee03dd91-8ebd-4268-9fbc-9017f73223b0-kube-api-access-5kd27\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:11.020816 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.020780 6507 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55ba2c05-8056-4ccc-beeb-118dff6c1875-plugins-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:11.099273 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.099238 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m"] Oct 29 23:32:11.099480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.099441 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" podUID="2d90e6ac-e91f-438e-ac8d-e4d027016263" containerName="istio-proxy" containerID="cri-o://5eeb9cfa30f268f40eec8f72e6f05e836c6ff7dd7e78ed0a268f7ae86068e9d1" gracePeriod=30 Oct 29 23:32:11.119168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.119133 6507 generic.go:356] "Generic (PLEG): container finished" podID="ee03dd91-8ebd-4268-9fbc-9017f73223b0" containerID="2a0e1f64eddc36d0540327e09c054b18c06a7007ab4dfd76302b7e9035d1d90e" exitCode=0 Oct 29 23:32:11.119316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.119192 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" Oct 29 23:32:11.119316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.119188 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" event={"ID":"ee03dd91-8ebd-4268-9fbc-9017f73223b0","Type":"ContainerDied","Data":"2a0e1f64eddc36d0540327e09c054b18c06a7007ab4dfd76302b7e9035d1d90e"} Oct 29 23:32:11.119316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.119253 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st" event={"ID":"ee03dd91-8ebd-4268-9fbc-9017f73223b0","Type":"ContainerDied","Data":"90300eb4368410880eba441b8d59fa6bb8a31da14c0484339a374adca1626072"} Oct 29 23:32:11.119316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.119287 6507 scope.go:122] "RemoveContainer" containerID="2a0e1f64eddc36d0540327e09c054b18c06a7007ab4dfd76302b7e9035d1d90e" Oct 29 23:32:11.120790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.120766 6507 generic.go:356] "Generic (PLEG): container finished" podID="55ba2c05-8056-4ccc-beeb-118dff6c1875" containerID="cb339fcf1cdfae9cb8c567d1841f8cb599fefedb8ee21fb04a65b53a850408eb" exitCode=0 Oct 29 23:32:11.120922 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.120807 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" Oct 29 23:32:11.120922 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.120812 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" event={"ID":"55ba2c05-8056-4ccc-beeb-118dff6c1875","Type":"ContainerDied","Data":"cb339fcf1cdfae9cb8c567d1841f8cb599fefedb8ee21fb04a65b53a850408eb"} Oct 29 23:32:11.120922 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.120841 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn" event={"ID":"55ba2c05-8056-4ccc-beeb-118dff6c1875","Type":"ContainerDied","Data":"80007441a067ec52134557fdb1251561669a2a1851c1e2ba511e02afc51157e4"} Oct 29 23:32:11.129141 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.129122 6507 scope.go:122] "RemoveContainer" containerID="2a0e1f64eddc36d0540327e09c054b18c06a7007ab4dfd76302b7e9035d1d90e" Oct 29 23:32:11.129439 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:32:11.129417 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0e1f64eddc36d0540327e09c054b18c06a7007ab4dfd76302b7e9035d1d90e\": container with ID starting with 2a0e1f64eddc36d0540327e09c054b18c06a7007ab4dfd76302b7e9035d1d90e not found: ID does not exist" containerID="2a0e1f64eddc36d0540327e09c054b18c06a7007ab4dfd76302b7e9035d1d90e" Oct 29 23:32:11.129681 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.129443 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0e1f64eddc36d0540327e09c054b18c06a7007ab4dfd76302b7e9035d1d90e"} err="failed to get container status \"2a0e1f64eddc36d0540327e09c054b18c06a7007ab4dfd76302b7e9035d1d90e\": rpc error: code = NotFound desc = could not find container \"2a0e1f64eddc36d0540327e09c054b18c06a7007ab4dfd76302b7e9035d1d90e\": container with ID starting with 2a0e1f64eddc36d0540327e09c054b18c06a7007ab4dfd76302b7e9035d1d90e not found: ID does not exist" Oct 29 23:32:11.129681 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.129491 6507 scope.go:122] "RemoveContainer" containerID="cb339fcf1cdfae9cb8c567d1841f8cb599fefedb8ee21fb04a65b53a850408eb" Oct 29 23:32:11.138787 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.138769 6507 scope.go:122] "RemoveContainer" containerID="cb339fcf1cdfae9cb8c567d1841f8cb599fefedb8ee21fb04a65b53a850408eb" Oct 29 23:32:11.139070 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:32:11.139051 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb339fcf1cdfae9cb8c567d1841f8cb599fefedb8ee21fb04a65b53a850408eb\": container with ID starting with cb339fcf1cdfae9cb8c567d1841f8cb599fefedb8ee21fb04a65b53a850408eb not found: ID does not exist" containerID="cb339fcf1cdfae9cb8c567d1841f8cb599fefedb8ee21fb04a65b53a850408eb" Oct 29 23:32:11.139123 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.139077 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb339fcf1cdfae9cb8c567d1841f8cb599fefedb8ee21fb04a65b53a850408eb"} err="failed to get container status \"cb339fcf1cdfae9cb8c567d1841f8cb599fefedb8ee21fb04a65b53a850408eb\": rpc error: code = NotFound desc = could not find container \"cb339fcf1cdfae9cb8c567d1841f8cb599fefedb8ee21fb04a65b53a850408eb\": container with ID starting with cb339fcf1cdfae9cb8c567d1841f8cb599fefedb8ee21fb04a65b53a850408eb not found: ID does not exist" Oct 29 23:32:11.168393 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.168366 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st"] Oct 29 23:32:11.173327 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.173299 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0b2ae9-6f988687f5-4b4st"] Oct 29 23:32:11.213678 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.212248 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn"] Oct 29 23:32:11.223572 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:11.223529 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0b2ae9-8469b6f8f8-7l7pn"] Oct 29 23:32:12.751590 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:12.751545 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ba2c05-8056-4ccc-beeb-118dff6c1875" path="/var/lib/kubelet/pods/55ba2c05-8056-4ccc-beeb-118dff6c1875/volumes" Oct 29 23:32:12.752107 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:12.752084 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee03dd91-8ebd-4268-9fbc-9017f73223b0" path="/var/lib/kubelet/pods/ee03dd91-8ebd-4268-9fbc-9017f73223b0/volumes" Oct 29 23:32:13.027391 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:13.027305 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j"] Oct 29 23:32:13.036331 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:13.036299 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdc4af09-9qm7j"] Oct 29 23:32:14.751900 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.751865 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cde2ebe4-9d4a-4b3f-b684-ed6dc850a469" path="/var/lib/kubelet/pods/cde2ebe4-9d4a-4b3f-b684-ed6dc850a469/volumes" Oct 29 23:32:14.976998 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.976957 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-765946bc59-mrxqg"] Oct 29 23:32:14.977463 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977440 6507 cpu_manager.go:414] "containerMap: removing container" podUID="6bf4f192-0296-4a17-abf7-b6a55b12f60c" containerName="istio-proxy" Oct 29 23:32:14.977624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977472 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf4f192-0296-4a17-abf7-b6a55b12f60c" containerName="istio-proxy" Oct 29 23:32:14.977624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977497 6507 cpu_manager.go:414] "containerMap: removing container" podUID="77a9bc98-cb47-403d-b1ea-824dde7ae578" containerName="payload-pre-processing" Oct 29 23:32:14.977624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977504 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77a9bc98-cb47-403d-b1ea-824dde7ae578" containerName="payload-pre-processing" Oct 29 23:32:14.977624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977516 6507 cpu_manager.go:414] "containerMap: removing container" podUID="80680fb1-ec38-4c40-8b92-ca49981bf54a" containerName="payload-processing" Oct 29 23:32:14.977624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977522 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80680fb1-ec38-4c40-8b92-ca49981bf54a" containerName="payload-processing" Oct 29 23:32:14.977624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977531 6507 cpu_manager.go:414] "containerMap: removing container" podUID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerName="istio-proxy" Oct 29 23:32:14.977624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977535 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerName="istio-proxy" Oct 29 23:32:14.977624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977544 6507 cpu_manager.go:414] "containerMap: removing container" podUID="22a425f6-b9c8-40a4-a403-fbd3a6d7baa4" containerName="revoke-keys" Oct 29 23:32:14.977624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977549 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a425f6-b9c8-40a4-a403-fbd3a6d7baa4" containerName="revoke-keys" Oct 29 23:32:14.977624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977561 6507 cpu_manager.go:414] "containerMap: removing container" podUID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerName="istio-proxy" Oct 29 23:32:14.977624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977571 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerName="istio-proxy" Oct 29 23:32:14.977624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977587 6507 cpu_manager.go:414] "containerMap: removing container" podUID="ee03dd91-8ebd-4268-9fbc-9017f73223b0" containerName="payload-pre-processing" Oct 29 23:32:14.977624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977595 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee03dd91-8ebd-4268-9fbc-9017f73223b0" containerName="payload-pre-processing" Oct 29 23:32:14.977624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977605 6507 cpu_manager.go:414] "containerMap: removing container" podUID="55ba2c05-8056-4ccc-beeb-118dff6c1875" containerName="payload-processing" Oct 29 23:32:14.977624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977613 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ba2c05-8056-4ccc-beeb-118dff6c1875" containerName="payload-processing" Oct 29 23:32:14.977624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977628 6507 cpu_manager.go:414] "containerMap: removing container" podUID="80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" containerName="istio-proxy" Oct 29 23:32:14.978245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977633 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" containerName="istio-proxy" Oct 29 23:32:14.978245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977728 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="77a9bc98-cb47-403d-b1ea-824dde7ae578" containerName="payload-pre-processing" Oct 29 23:32:14.978245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977751 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="80680fb1-ec38-4c40-8b92-ca49981bf54a" containerName="payload-processing" Oct 29 23:32:14.978245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977761 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerName="istio-proxy" Oct 29 23:32:14.978245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977771 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a425f6-b9c8-40a4-a403-fbd3a6d7baa4" containerName="revoke-keys" Oct 29 23:32:14.978245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977785 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee03dd91-8ebd-4268-9fbc-9017f73223b0" containerName="payload-pre-processing" Oct 29 23:32:14.978245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977795 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ba2c05-8056-4ccc-beeb-118dff6c1875" containerName="payload-processing" Oct 29 23:32:14.978245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.977806 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c9ffd0-41a3-47a2-84d2-5b70d9b99e5c" containerName="istio-proxy" Oct 29 23:32:14.982718 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:14.982694 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-765946bc59-mrxqg" Oct 29 23:32:15.003952 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.003887 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 29 23:32:15.003952 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.003914 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-f5jqh\"" Oct 29 23:32:15.004100 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.003973 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 29 23:32:15.004100 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.004029 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 29 23:32:15.008273 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.008244 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-765946bc59-mrxqg"] Oct 29 23:32:15.062114 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.062076 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d49gq\" (UniqueName: \"kubernetes.io/projected/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-kube-api-access-d49gq\") pod \"maas-controller-765946bc59-mrxqg\" (UID: \"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466\") " pod="opendatahub/maas-controller-765946bc59-mrxqg" Oct 29 23:32:15.062286 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.062195 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-webhook-cert\") pod \"maas-controller-765946bc59-mrxqg\" (UID: \"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466\") " pod="opendatahub/maas-controller-765946bc59-mrxqg" Oct 29 23:32:15.062286 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.062229 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-metrics-certs\") pod \"maas-controller-765946bc59-mrxqg\" (UID: \"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466\") " pod="opendatahub/maas-controller-765946bc59-mrxqg" Oct 29 23:32:15.163064 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.163018 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49gq\" (UniqueName: \"kubernetes.io/projected/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-kube-api-access-d49gq\") pod \"maas-controller-765946bc59-mrxqg\" (UID: \"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466\") " pod="opendatahub/maas-controller-765946bc59-mrxqg" Oct 29 23:32:15.163245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.163106 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-webhook-cert\") pod \"maas-controller-765946bc59-mrxqg\" (UID: \"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466\") " pod="opendatahub/maas-controller-765946bc59-mrxqg" Oct 29 23:32:15.163245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.163124 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-metrics-certs\") pod \"maas-controller-765946bc59-mrxqg\" (UID: \"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466\") " pod="opendatahub/maas-controller-765946bc59-mrxqg" Oct 29 23:32:15.165617 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.165592 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-webhook-cert\") pod \"maas-controller-765946bc59-mrxqg\" (UID: \"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466\") " pod="opendatahub/maas-controller-765946bc59-mrxqg" Oct 29 23:32:15.165725 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.165625 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-metrics-certs\") pod \"maas-controller-765946bc59-mrxqg\" (UID: \"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466\") " pod="opendatahub/maas-controller-765946bc59-mrxqg" Oct 29 23:32:15.179379 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.179353 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49gq\" (UniqueName: \"kubernetes.io/projected/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-kube-api-access-d49gq\") pod \"maas-controller-765946bc59-mrxqg\" (UID: \"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466\") " pod="opendatahub/maas-controller-765946bc59-mrxqg" Oct 29 23:32:15.222571 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:32:15.222537 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:32:15.296186 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.296104 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-765946bc59-mrxqg" Oct 29 23:32:15.594670 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.594595 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-765946bc59-mrxqg"] Oct 29 23:32:15.598009 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:32:15.597975 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d83ecc_9bc1_4d67_8f6c_43b1e0c13466.slice/crio-b9331a04b2bcc835188725a0fbf55d138d6f57bad89b31163739b2b6b93caa4c WatchSource:0}: Error finding container b9331a04b2bcc835188725a0fbf55d138d6f57bad89b31163739b2b6b93caa4c: Status 404 returned error can't find the container with id b9331a04b2bcc835188725a0fbf55d138d6f57bad89b31163739b2b6b93caa4c Oct 29 23:32:15.599327 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:15.599307 6507 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:32:16.167217 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.167130 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-765946bc59-mrxqg" event={"ID":"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466","Type":"ContainerStarted","Data":"b9331a04b2bcc835188725a0fbf55d138d6f57bad89b31163739b2b6b93caa4c"} Oct 29 23:32:16.477438 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.477405 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:32:16.580497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.578520 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-data\") pod \"2d90e6ac-e91f-438e-ac8d-e4d027016263\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " Oct 29 23:32:16.580497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.578572 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-credential-socket\") pod \"2d90e6ac-e91f-438e-ac8d-e4d027016263\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " Oct 29 23:32:16.580497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.578609 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-workload-socket\") pod \"2d90e6ac-e91f-438e-ac8d-e4d027016263\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " Oct 29 23:32:16.580497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.578653 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-workload-certs\") pod \"2d90e6ac-e91f-438e-ac8d-e4d027016263\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " Oct 29 23:32:16.580497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.578736 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d90e6ac-e91f-438e-ac8d-e4d027016263-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d90e6ac-e91f-438e-ac8d-e4d027016263-istiod-ca-cert\") pod \"2d90e6ac-e91f-438e-ac8d-e4d027016263\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " Oct 29 23:32:16.580497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.578792 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d90e6ac-e91f-438e-ac8d-e4d027016263-kube-api-access-gnxrj\" (UniqueName: \"kubernetes.io/projected/2d90e6ac-e91f-438e-ac8d-e4d027016263-kube-api-access-gnxrj\") pod \"2d90e6ac-e91f-438e-ac8d-e4d027016263\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " Oct 29 23:32:16.580497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.578861 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-envoy\") pod \"2d90e6ac-e91f-438e-ac8d-e4d027016263\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " Oct 29 23:32:16.580497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.578905 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-token\" (UniqueName: \"kubernetes.io/projected/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-token\") pod \"2d90e6ac-e91f-438e-ac8d-e4d027016263\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " Oct 29 23:32:16.580497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.578940 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-podinfo\") pod \"2d90e6ac-e91f-438e-ac8d-e4d027016263\" (UID: \"2d90e6ac-e91f-438e-ac8d-e4d027016263\") " Oct 29 23:32:16.581387 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.581218 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-credential-socket" pod "2d90e6ac-e91f-438e-ac8d-e4d027016263" (UID: "2d90e6ac-e91f-438e-ac8d-e4d027016263"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:16.581387 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.581306 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-workload-socket" pod "2d90e6ac-e91f-438e-ac8d-e4d027016263" (UID: "2d90e6ac-e91f-438e-ac8d-e4d027016263"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:16.582354 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.581384 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-workload-certs" pod "2d90e6ac-e91f-438e-ac8d-e4d027016263" (UID: "2d90e6ac-e91f-438e-ac8d-e4d027016263"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:16.582354 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.582028 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d90e6ac-e91f-438e-ac8d-e4d027016263-istiod-ca-cert" pod "2d90e6ac-e91f-438e-ac8d-e4d027016263" (UID: "2d90e6ac-e91f-438e-ac8d-e4d027016263"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:16.582354 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.582174 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-data" pod "2d90e6ac-e91f-438e-ac8d-e4d027016263" (UID: "2d90e6ac-e91f-438e-ac8d-e4d027016263"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:16.587747 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.587682 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-envoy" pod "2d90e6ac-e91f-438e-ac8d-e4d027016263" (UID: "2d90e6ac-e91f-438e-ac8d-e4d027016263"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:16.600514 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.600423 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d90e6ac-e91f-438e-ac8d-e4d027016263-kube-api-access-gnxrj" pod "2d90e6ac-e91f-438e-ac8d-e4d027016263" (UID: "2d90e6ac-e91f-438e-ac8d-e4d027016263"). InnerVolumeSpecName "kube-api-access-gnxrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:16.604533 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.604433 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-podinfo" pod "2d90e6ac-e91f-438e-ac8d-e4d027016263" (UID: "2d90e6ac-e91f-438e-ac8d-e4d027016263"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:32:16.610864 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.610829 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-token" pod "2d90e6ac-e91f-438e-ac8d-e4d027016263" (UID: "2d90e6ac-e91f-438e-ac8d-e4d027016263"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:16.681035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.680852 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d90e6ac-e91f-438e-ac8d-e4d027016263-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:16.681035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.680889 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnxrj\" (UniqueName: \"kubernetes.io/projected/2d90e6ac-e91f-438e-ac8d-e4d027016263-kube-api-access-gnxrj\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:16.681035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.680905 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:16.681035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.680921 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:16.681035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.680936 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:16.681035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.680950 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:16.681035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.680963 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:16.681035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.680977 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:16.681035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:16.680991 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d90e6ac-e91f-438e-ac8d-e4d027016263-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:17.192357 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:17.192324 6507 generic.go:356] "Generic (PLEG): container finished" podID="2d90e6ac-e91f-438e-ac8d-e4d027016263" containerID="5eeb9cfa30f268f40eec8f72e6f05e836c6ff7dd7e78ed0a268f7ae86068e9d1" exitCode=0 Oct 29 23:32:17.193508 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:17.193473 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" Oct 29 23:32:17.194206 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:17.194180 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" event={"ID":"2d90e6ac-e91f-438e-ac8d-e4d027016263","Type":"ContainerDied","Data":"5eeb9cfa30f268f40eec8f72e6f05e836c6ff7dd7e78ed0a268f7ae86068e9d1"} Oct 29 23:32:17.194331 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:17.194257 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m" event={"ID":"2d90e6ac-e91f-438e-ac8d-e4d027016263","Type":"ContainerDied","Data":"3593935cae80e5cee170e596e4f3e12cbd6b41880dc980e1900c2c64e3b27c24"} Oct 29 23:32:17.194331 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:17.194293 6507 scope.go:122] "RemoveContainer" containerID="5eeb9cfa30f268f40eec8f72e6f05e836c6ff7dd7e78ed0a268f7ae86068e9d1" Oct 29 23:32:17.272974 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:17.272950 6507 scope.go:122] "RemoveContainer" containerID="5eeb9cfa30f268f40eec8f72e6f05e836c6ff7dd7e78ed0a268f7ae86068e9d1" Oct 29 23:32:17.304670 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:32:17.304625 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eeb9cfa30f268f40eec8f72e6f05e836c6ff7dd7e78ed0a268f7ae86068e9d1\": container with ID starting with 5eeb9cfa30f268f40eec8f72e6f05e836c6ff7dd7e78ed0a268f7ae86068e9d1 not found: ID does not exist" containerID="5eeb9cfa30f268f40eec8f72e6f05e836c6ff7dd7e78ed0a268f7ae86068e9d1" Oct 29 23:32:17.304926 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:17.304890 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eeb9cfa30f268f40eec8f72e6f05e836c6ff7dd7e78ed0a268f7ae86068e9d1"} err="failed to get container status \"5eeb9cfa30f268f40eec8f72e6f05e836c6ff7dd7e78ed0a268f7ae86068e9d1\": rpc error: code = NotFound desc = could not find container \"5eeb9cfa30f268f40eec8f72e6f05e836c6ff7dd7e78ed0a268f7ae86068e9d1\": container with ID starting with 5eeb9cfa30f268f40eec8f72e6f05e836c6ff7dd7e78ed0a268f7ae86068e9d1 not found: ID does not exist" Oct 29 23:32:17.305100 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:17.305084 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m"] Oct 29 23:32:17.337481 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:17.333655 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0b2ae9-openshift-default-767d4d54bf-kqd6m"] Oct 29 23:32:18.780834 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:18.780416 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d90e6ac-e91f-438e-ac8d-e4d027016263" path="/var/lib/kubelet/pods/2d90e6ac-e91f-438e-ac8d-e4d027016263/volumes" Oct 29 23:32:19.002316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:19.002038 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-9phht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:32:19.002316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:19.002115 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-9phht" podUID="f116d226-bd4f-41bd-aba4-06958eccb990" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:32:21.265474 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:21.265418 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-765946bc59-mrxqg" event={"ID":"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466","Type":"ContainerStarted","Data":"59e66c3477262ed26fcd75d1d30d8f8af89084ed8ff1c1c8a73bd1c9bc32d6de"} Oct 29 23:32:21.265894 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:21.265606 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-765946bc59-mrxqg" Oct 29 23:32:21.304162 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:21.304100 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-765946bc59-mrxqg" podStartSLOduration=2.436731184 podStartE2EDuration="7.304082814s" totalImagesPullingTime="4.86735163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:21.301922178 +0000 UTC m=+1110.664926900" watchObservedRunningTime="2026-10-29 23:32:21.304082814 +0000 UTC m=+1110.667087536" Oct 29 23:32:32.274612 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:32.274577 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-765946bc59-mrxqg" Oct 29 23:32:38.671315 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.671261 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2"] Oct 29 23:32:38.671861 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.671770 6507 cpu_manager.go:414] "containerMap: removing container" podUID="2d90e6ac-e91f-438e-ac8d-e4d027016263" containerName="istio-proxy" Oct 29 23:32:38.671861 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.671786 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d90e6ac-e91f-438e-ac8d-e4d027016263" containerName="istio-proxy" Oct 29 23:32:38.671861 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.671855 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac751e7-bb4e-4bf7-81f2-26fee5a478f3" containerName="istio-proxy" Oct 29 23:32:38.671968 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.671864 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d90e6ac-e91f-438e-ac8d-e4d027016263" containerName="istio-proxy" Oct 29 23:32:38.675087 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.675068 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2" Oct 29 23:32:38.687382 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.687360 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:32:38.696866 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.696841 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:32:38.696866 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.696850 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:32:38.697049 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.696862 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:32:38.699392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.699371 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2"] Oct 29 23:32:38.774297 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.774259 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2711c384-96b8-4a4b-9e40-ab39da492c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-pd8t2\" (UID: \"2711c384-96b8-4a4b-9e40-ab39da492c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2" Oct 29 23:32:38.774511 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.774414 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf5pv\" (UniqueName: \"kubernetes.io/projected/2711c384-96b8-4a4b-9e40-ab39da492c26-kube-api-access-jf5pv\") pod \"maas-api-revoke-sub-85237e45-pd8t2\" (UID: \"2711c384-96b8-4a4b-9e40-ab39da492c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2" Oct 29 23:32:38.774511 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.774448 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2711c384-96b8-4a4b-9e40-ab39da492c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-pd8t2\" (UID: \"2711c384-96b8-4a4b-9e40-ab39da492c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2" Oct 29 23:32:38.875618 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.875580 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf5pv\" (UniqueName: \"kubernetes.io/projected/2711c384-96b8-4a4b-9e40-ab39da492c26-kube-api-access-jf5pv\") pod \"maas-api-revoke-sub-85237e45-pd8t2\" (UID: \"2711c384-96b8-4a4b-9e40-ab39da492c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2" Oct 29 23:32:38.875618 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.875623 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2711c384-96b8-4a4b-9e40-ab39da492c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-pd8t2\" (UID: \"2711c384-96b8-4a4b-9e40-ab39da492c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2" Oct 29 23:32:38.875851 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.875682 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2711c384-96b8-4a4b-9e40-ab39da492c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-pd8t2\" (UID: \"2711c384-96b8-4a4b-9e40-ab39da492c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2" Oct 29 23:32:38.876312 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.876288 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2711c384-96b8-4a4b-9e40-ab39da492c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-pd8t2\" (UID: \"2711c384-96b8-4a4b-9e40-ab39da492c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2" Oct 29 23:32:38.878030 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.878005 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2711c384-96b8-4a4b-9e40-ab39da492c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-pd8t2\" (UID: \"2711c384-96b8-4a4b-9e40-ab39da492c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2" Oct 29 23:32:38.895207 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.895178 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf5pv\" (UniqueName: \"kubernetes.io/projected/2711c384-96b8-4a4b-9e40-ab39da492c26-kube-api-access-jf5pv\") pod \"maas-api-revoke-sub-85237e45-pd8t2\" (UID: \"2711c384-96b8-4a4b-9e40-ab39da492c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2" Oct 29 23:32:38.988303 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:38.988271 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2" Oct 29 23:32:39.288587 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:39.288490 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2"] Oct 29 23:32:39.292061 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:32:39.292033 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2711c384_96b8_4a4b_9e40_ab39da492c26.slice/crio-78181071f8c740d0f10d01c0b162c515e236b1ade932c1ea8af15fb2bda22ebe WatchSource:0}: Error finding container 78181071f8c740d0f10d01c0b162c515e236b1ade932c1ea8af15fb2bda22ebe: Status 404 returned error can't find the container with id 78181071f8c740d0f10d01c0b162c515e236b1ade932c1ea8af15fb2bda22ebe Oct 29 23:32:39.379007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:39.378971 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2" event={"ID":"2711c384-96b8-4a4b-9e40-ab39da492c26","Type":"ContainerStarted","Data":"78181071f8c740d0f10d01c0b162c515e236b1ade932c1ea8af15fb2bda22ebe"} Oct 29 23:32:40.388807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:40.388772 6507 generic.go:356] "Generic (PLEG): container finished" podID="2711c384-96b8-4a4b-9e40-ab39da492c26" containerID="9ac5a40ba91772ef032d84e9ad998e026bcf0043f4e5db769debe97767a64ef8" exitCode=0 Oct 29 23:32:40.389289 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:40.388945 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2" event={"ID":"2711c384-96b8-4a4b-9e40-ab39da492c26","Type":"ContainerDied","Data":"9ac5a40ba91772ef032d84e9ad998e026bcf0043f4e5db769debe97767a64ef8"} Oct 29 23:32:41.537143 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:41.537113 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2" Oct 29 23:32:41.605407 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:41.605364 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2711c384-96b8-4a4b-9e40-ab39da492c26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2711c384-96b8-4a4b-9e40-ab39da492c26-maas-api-xks-ca\") pod \"2711c384-96b8-4a4b-9e40-ab39da492c26\" (UID: \"2711c384-96b8-4a4b-9e40-ab39da492c26\") " Oct 29 23:32:41.605626 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:41.605434 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2711c384-96b8-4a4b-9e40-ab39da492c26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2711c384-96b8-4a4b-9e40-ab39da492c26-maas-api-openshift-ca\") pod \"2711c384-96b8-4a4b-9e40-ab39da492c26\" (UID: \"2711c384-96b8-4a4b-9e40-ab39da492c26\") " Oct 29 23:32:41.605626 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:41.605572 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2711c384-96b8-4a4b-9e40-ab39da492c26-kube-api-access-jf5pv\" (UniqueName: \"kubernetes.io/projected/2711c384-96b8-4a4b-9e40-ab39da492c26-kube-api-access-jf5pv\") pod \"2711c384-96b8-4a4b-9e40-ab39da492c26\" (UID: \"2711c384-96b8-4a4b-9e40-ab39da492c26\") " Oct 29 23:32:41.605897 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:41.605863 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2711c384-96b8-4a4b-9e40-ab39da492c26-maas-api-openshift-ca" pod "2711c384-96b8-4a4b-9e40-ab39da492c26" (UID: "2711c384-96b8-4a4b-9e40-ab39da492c26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:41.606317 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:41.606288 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2711c384-96b8-4a4b-9e40-ab39da492c26-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:41.608163 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:41.608128 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2711c384-96b8-4a4b-9e40-ab39da492c26-maas-api-xks-ca" pod "2711c384-96b8-4a4b-9e40-ab39da492c26" (UID: "2711c384-96b8-4a4b-9e40-ab39da492c26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:41.608441 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:41.608413 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2711c384-96b8-4a4b-9e40-ab39da492c26-kube-api-access-jf5pv" pod "2711c384-96b8-4a4b-9e40-ab39da492c26" (UID: "2711c384-96b8-4a4b-9e40-ab39da492c26"). InnerVolumeSpecName "kube-api-access-jf5pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:41.707203 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:41.707169 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2711c384-96b8-4a4b-9e40-ab39da492c26-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:41.707203 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:41.707200 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf5pv\" (UniqueName: \"kubernetes.io/projected/2711c384-96b8-4a4b-9e40-ab39da492c26-kube-api-access-jf5pv\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:42.403814 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:42.403786 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2" Oct 29 23:32:42.403985 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:42.403820 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2" event={"ID":"2711c384-96b8-4a4b-9e40-ab39da492c26","Type":"ContainerDied","Data":"78181071f8c740d0f10d01c0b162c515e236b1ade932c1ea8af15fb2bda22ebe"} Oct 29 23:32:42.403985 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:42.403870 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78181071f8c740d0f10d01c0b162c515e236b1ade932c1ea8af15fb2bda22ebe" Oct 29 23:32:45.224548 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:32:45.224503 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:32:50.398514 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:50.398475 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-765946bc59-mrxqg"] Oct 29 23:32:50.399124 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:50.398736 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-765946bc59-mrxqg" podUID="18d83ecc-9bc1-4d67-8f6c-43b1e0c13466" containerName="manager" containerID="cri-o://59e66c3477262ed26fcd75d1d30d8f8af89084ed8ff1c1c8a73bd1c9bc32d6de" gracePeriod=30 Oct 29 23:32:50.636161 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:50.636131 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-765946bc59-mrxqg" Oct 29 23:32:50.801958 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:50.801394 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-metrics-certs\") pod \"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466\" (UID: \"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466\") " Oct 29 23:32:50.801958 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:50.801476 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-webhook-cert\" (UniqueName: \"kubernetes.io/secret/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-webhook-cert\") pod \"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466\" (UID: \"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466\") " Oct 29 23:32:50.801958 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:50.801554 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-kube-api-access-d49gq\" (UniqueName: \"kubernetes.io/projected/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-kube-api-access-d49gq\") pod \"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466\" (UID: \"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466\") " Oct 29 23:32:50.806527 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:50.804934 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-webhook-cert" pod "18d83ecc-9bc1-4d67-8f6c-43b1e0c13466" (UID: "18d83ecc-9bc1-4d67-8f6c-43b1e0c13466"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:50.806527 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:50.804958 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-metrics-certs" pod "18d83ecc-9bc1-4d67-8f6c-43b1e0c13466" (UID: "18d83ecc-9bc1-4d67-8f6c-43b1e0c13466"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:50.809415 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:50.809388 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-kube-api-access-d49gq" pod "18d83ecc-9bc1-4d67-8f6c-43b1e0c13466" (UID: "18d83ecc-9bc1-4d67-8f6c-43b1e0c13466"). InnerVolumeSpecName "kube-api-access-d49gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:50.903492 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:50.903432 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d49gq\" (UniqueName: \"kubernetes.io/projected/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-kube-api-access-d49gq\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:50.903658 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:50.903525 6507 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-metrics-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:50.903658 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:50.903554 6507 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466-webhook-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:32:51.460899 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:51.460861 6507 generic.go:356] "Generic (PLEG): container finished" podID="18d83ecc-9bc1-4d67-8f6c-43b1e0c13466" containerID="59e66c3477262ed26fcd75d1d30d8f8af89084ed8ff1c1c8a73bd1c9bc32d6de" exitCode=0 Oct 29 23:32:51.461315 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:51.460922 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-765946bc59-mrxqg" Oct 29 23:32:51.461315 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:51.460947 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-765946bc59-mrxqg" event={"ID":"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466","Type":"ContainerDied","Data":"59e66c3477262ed26fcd75d1d30d8f8af89084ed8ff1c1c8a73bd1c9bc32d6de"} Oct 29 23:32:51.461315 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:51.461029 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-765946bc59-mrxqg" event={"ID":"18d83ecc-9bc1-4d67-8f6c-43b1e0c13466","Type":"ContainerDied","Data":"b9331a04b2bcc835188725a0fbf55d138d6f57bad89b31163739b2b6b93caa4c"} Oct 29 23:32:51.461315 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:51.461052 6507 scope.go:122] "RemoveContainer" containerID="59e66c3477262ed26fcd75d1d30d8f8af89084ed8ff1c1c8a73bd1c9bc32d6de" Oct 29 23:32:51.470550 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:51.470530 6507 scope.go:122] "RemoveContainer" containerID="59e66c3477262ed26fcd75d1d30d8f8af89084ed8ff1c1c8a73bd1c9bc32d6de" Oct 29 23:32:51.470812 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:32:51.470793 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e66c3477262ed26fcd75d1d30d8f8af89084ed8ff1c1c8a73bd1c9bc32d6de\": container with ID starting with 59e66c3477262ed26fcd75d1d30d8f8af89084ed8ff1c1c8a73bd1c9bc32d6de not found: ID does not exist" containerID="59e66c3477262ed26fcd75d1d30d8f8af89084ed8ff1c1c8a73bd1c9bc32d6de" Oct 29 23:32:51.470874 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:51.470824 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e66c3477262ed26fcd75d1d30d8f8af89084ed8ff1c1c8a73bd1c9bc32d6de"} err="failed to get container status \"59e66c3477262ed26fcd75d1d30d8f8af89084ed8ff1c1c8a73bd1c9bc32d6de\": rpc error: code = NotFound desc = could not find container \"59e66c3477262ed26fcd75d1d30d8f8af89084ed8ff1c1c8a73bd1c9bc32d6de\": container with ID starting with 59e66c3477262ed26fcd75d1d30d8f8af89084ed8ff1c1c8a73bd1c9bc32d6de not found: ID does not exist" Oct 29 23:32:51.503955 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:51.503927 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-765946bc59-mrxqg"] Oct 29 23:32:51.512867 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:51.512845 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-765946bc59-mrxqg"] Oct 29 23:32:52.605206 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.605172 6507 scope.go:122] "RemoveContainer" containerID="872f9a79210858801240386626f0be7ef4059ebde4cba354019228222b475021" Oct 29 23:32:52.615068 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.615047 6507 scope.go:122] "RemoveContainer" containerID="8b899ec98461060347fa08bcedfaea4b5145dbae8e54be5a847057c0025b078e" Oct 29 23:32:52.623941 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.623921 6507 scope.go:122] "RemoveContainer" containerID="9882f38ea9565ce8f9817aad3ea54a678a5f6e613c75a1a488ba461a8d2782f5" Oct 29 23:32:52.634565 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.634542 6507 scope.go:122] "RemoveContainer" containerID="17a2c7627d599f637ec67437c291c995bf2a0e4387f7ac02f95a8110f42e7e8c" Oct 29 23:32:52.643407 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.643387 6507 scope.go:122] "RemoveContainer" containerID="8a2dbef67968f68c3c77270765bc12ed56e9adb5b5bec321ea46dab8a9f8bce4" Oct 29 23:32:52.651215 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.651196 6507 scope.go:122] "RemoveContainer" containerID="504e990d320bdaaaebc7d5b102acb38a4062bcfcaf4e8139c0320f23d937e855" Oct 29 23:32:52.659114 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.659087 6507 scope.go:122] "RemoveContainer" containerID="d287bcfeef48953e9aa0f5ef32690e849c0c61143f951c145894514f46b193ad" Oct 29 23:32:52.666782 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.666762 6507 scope.go:122] "RemoveContainer" containerID="d388a055e2959ad9d669cad55b5e2849cf7296ef936dec531d6e1e1c35770c0b" Oct 29 23:32:52.674653 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.674630 6507 scope.go:122] "RemoveContainer" containerID="6f4be05590aa9dd09872aaaad9d8486badee0228758e3593dae79b0c5ab28f25" Oct 29 23:32:52.682121 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.682102 6507 scope.go:122] "RemoveContainer" containerID="be7faa19e4833cbe392ba2efe492a28354a939866d02a9cc3bef4ac58636cb5a" Oct 29 23:32:52.740843 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.740817 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-765946bc59-h74cr"] Oct 29 23:32:52.741192 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.741180 6507 cpu_manager.go:414] "containerMap: removing container" podUID="18d83ecc-9bc1-4d67-8f6c-43b1e0c13466" containerName="manager" Oct 29 23:32:52.741235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.741194 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d83ecc-9bc1-4d67-8f6c-43b1e0c13466" containerName="manager" Oct 29 23:32:52.741235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.741226 6507 cpu_manager.go:414] "containerMap: removing container" podUID="2711c384-96b8-4a4b-9e40-ab39da492c26" containerName="revoke-keys" Oct 29 23:32:52.741235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.741232 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2711c384-96b8-4a4b-9e40-ab39da492c26" containerName="revoke-keys" Oct 29 23:32:52.741336 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.741284 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d83ecc-9bc1-4d67-8f6c-43b1e0c13466" containerName="manager" Oct 29 23:32:52.741336 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.741299 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="2711c384-96b8-4a4b-9e40-ab39da492c26" containerName="revoke-keys" Oct 29 23:32:52.746734 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.746714 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-765946bc59-h74cr" Oct 29 23:32:52.753883 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.753655 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d83ecc-9bc1-4d67-8f6c-43b1e0c13466" path="/var/lib/kubelet/pods/18d83ecc-9bc1-4d67-8f6c-43b1e0c13466/volumes" Oct 29 23:32:52.758852 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.758826 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 29 23:32:52.769618 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.768417 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 29 23:32:52.769618 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.768547 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 29 23:32:52.769618 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.769587 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-f5jqh\"" Oct 29 23:32:52.771695 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.771676 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-765946bc59-h74cr"] Oct 29 23:32:52.922554 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.922421 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9c239c8-1cc2-408e-bfda-95be8f2da85f-webhook-cert\") pod \"maas-controller-765946bc59-h74cr\" (UID: \"b9c239c8-1cc2-408e-bfda-95be8f2da85f\") " pod="opendatahub/maas-controller-765946bc59-h74cr" Oct 29 23:32:52.922554 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.922489 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c239c8-1cc2-408e-bfda-95be8f2da85f-metrics-certs\") pod \"maas-controller-765946bc59-h74cr\" (UID: \"b9c239c8-1cc2-408e-bfda-95be8f2da85f\") " pod="opendatahub/maas-controller-765946bc59-h74cr" Oct 29 23:32:52.922803 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:52.922630 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jnjh\" (UniqueName: \"kubernetes.io/projected/b9c239c8-1cc2-408e-bfda-95be8f2da85f-kube-api-access-7jnjh\") pod \"maas-controller-765946bc59-h74cr\" (UID: \"b9c239c8-1cc2-408e-bfda-95be8f2da85f\") " pod="opendatahub/maas-controller-765946bc59-h74cr" Oct 29 23:32:53.023742 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:53.023710 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9c239c8-1cc2-408e-bfda-95be8f2da85f-webhook-cert\") pod \"maas-controller-765946bc59-h74cr\" (UID: \"b9c239c8-1cc2-408e-bfda-95be8f2da85f\") " pod="opendatahub/maas-controller-765946bc59-h74cr" Oct 29 23:32:53.023902 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:53.023763 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c239c8-1cc2-408e-bfda-95be8f2da85f-metrics-certs\") pod \"maas-controller-765946bc59-h74cr\" (UID: \"b9c239c8-1cc2-408e-bfda-95be8f2da85f\") " pod="opendatahub/maas-controller-765946bc59-h74cr" Oct 29 23:32:53.023902 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:53.023810 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jnjh\" (UniqueName: \"kubernetes.io/projected/b9c239c8-1cc2-408e-bfda-95be8f2da85f-kube-api-access-7jnjh\") pod \"maas-controller-765946bc59-h74cr\" (UID: \"b9c239c8-1cc2-408e-bfda-95be8f2da85f\") " pod="opendatahub/maas-controller-765946bc59-h74cr" Oct 29 23:32:53.026228 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:53.026195 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9c239c8-1cc2-408e-bfda-95be8f2da85f-webhook-cert\") pod \"maas-controller-765946bc59-h74cr\" (UID: \"b9c239c8-1cc2-408e-bfda-95be8f2da85f\") " pod="opendatahub/maas-controller-765946bc59-h74cr" Oct 29 23:32:53.026228 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:53.026215 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c239c8-1cc2-408e-bfda-95be8f2da85f-metrics-certs\") pod \"maas-controller-765946bc59-h74cr\" (UID: \"b9c239c8-1cc2-408e-bfda-95be8f2da85f\") " pod="opendatahub/maas-controller-765946bc59-h74cr" Oct 29 23:32:53.040348 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:53.040321 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jnjh\" (UniqueName: \"kubernetes.io/projected/b9c239c8-1cc2-408e-bfda-95be8f2da85f-kube-api-access-7jnjh\") pod \"maas-controller-765946bc59-h74cr\" (UID: \"b9c239c8-1cc2-408e-bfda-95be8f2da85f\") " pod="opendatahub/maas-controller-765946bc59-h74cr" Oct 29 23:32:53.061987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:53.061959 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-765946bc59-h74cr" Oct 29 23:32:53.368822 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:53.368785 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-765946bc59-h74cr"] Oct 29 23:32:53.372252 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:32:53.372204 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c239c8_1cc2_408e_bfda_95be8f2da85f.slice/crio-e51e45a29b14470eaface1b3b9cbd7593db37e0546ae89c612cf58d211726239 WatchSource:0}: Error finding container e51e45a29b14470eaface1b3b9cbd7593db37e0546ae89c612cf58d211726239: Status 404 returned error can't find the container with id e51e45a29b14470eaface1b3b9cbd7593db37e0546ae89c612cf58d211726239 Oct 29 23:32:53.476330 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:53.476292 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-765946bc59-h74cr" event={"ID":"b9c239c8-1cc2-408e-bfda-95be8f2da85f","Type":"ContainerStarted","Data":"e51e45a29b14470eaface1b3b9cbd7593db37e0546ae89c612cf58d211726239"} Oct 29 23:32:54.484398 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:54.484353 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-765946bc59-h74cr" event={"ID":"b9c239c8-1cc2-408e-bfda-95be8f2da85f","Type":"ContainerStarted","Data":"819c51aedf09efaa49df03d16028aa15aa25ae86b2f37808914978bf90b9945e"} Oct 29 23:32:54.484823 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:54.484545 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-765946bc59-h74cr" Oct 29 23:32:54.533828 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:54.533763 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-765946bc59-h74cr" podStartSLOduration=2.143821804 podStartE2EDuration="2.533750451s" totalImagesPullingTime="389.928647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:54.533024618 +0000 UTC m=+1143.896029340" watchObservedRunningTime="2026-10-29 23:32:54.533750451 +0000 UTC m=+1143.896755190" Oct 29 23:32:56.025482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:56.025433 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5"] Oct 29 23:32:56.034682 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:56.034645 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ad5379-pkjj5"] Oct 29 23:32:56.751290 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:32:56.751254 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e700ea3-be62-40d2-b685-99dd65a4289d" path="/var/lib/kubelet/pods/4e700ea3-be62-40d2-b685-99dd65a4289d/volumes" Oct 29 23:33:05.493041 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:05.493005 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-765946bc59-h74cr" Oct 29 23:33:15.223251 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:33:15.223214 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:33:21.399846 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:21.399808 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:33:24.040621 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:24.040582 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx"] Oct 29 23:33:24.051482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:24.044812 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a18c96f4-q8nhx"] Oct 29 23:33:24.752636 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:24.752600 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4410de-1052-48de-8eb5-9380c75b35a8" path="/var/lib/kubelet/pods/ec4410de-1052-48de-8eb5-9380c75b35a8/volumes" Oct 29 23:33:27.413711 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.413673 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p"] Oct 29 23:33:27.418765 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.418744 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" Oct 29 23:33:27.445921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.445788 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:33:27.445921 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.445893 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:33:27.446371 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.446352 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:33:27.446775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.446758 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:33:27.471970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.471936 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p"] Oct 29 23:33:27.484063 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.484031 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-t4p6p\" (UID: \"c109ab0d-5a84-45d3-bb8f-9e29a34d416a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" Oct 29 23:33:27.484261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.484099 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92bz2\" (UniqueName: \"kubernetes.io/projected/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-kube-api-access-92bz2\") pod \"maas-api-revoke-sub-c1e271a9-t4p6p\" (UID: \"c109ab0d-5a84-45d3-bb8f-9e29a34d416a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" Oct 29 23:33:27.484261 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.484239 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-t4p6p\" (UID: \"c109ab0d-5a84-45d3-bb8f-9e29a34d416a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" Oct 29 23:33:27.584955 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.584919 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92bz2\" (UniqueName: \"kubernetes.io/projected/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-kube-api-access-92bz2\") pod \"maas-api-revoke-sub-c1e271a9-t4p6p\" (UID: \"c109ab0d-5a84-45d3-bb8f-9e29a34d416a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" Oct 29 23:33:27.585165 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.585002 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-t4p6p\" (UID: \"c109ab0d-5a84-45d3-bb8f-9e29a34d416a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" Oct 29 23:33:27.585165 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.585075 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-t4p6p\" (UID: \"c109ab0d-5a84-45d3-bb8f-9e29a34d416a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" Oct 29 23:33:27.585723 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.585697 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-t4p6p\" (UID: \"c109ab0d-5a84-45d3-bb8f-9e29a34d416a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" Oct 29 23:33:27.587491 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.587446 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-t4p6p\" (UID: \"c109ab0d-5a84-45d3-bb8f-9e29a34d416a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" Oct 29 23:33:27.602066 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.602042 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92bz2\" (UniqueName: \"kubernetes.io/projected/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-kube-api-access-92bz2\") pod \"maas-api-revoke-sub-c1e271a9-t4p6p\" (UID: \"c109ab0d-5a84-45d3-bb8f-9e29a34d416a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" Oct 29 23:33:27.732431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:27.732391 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" Oct 29 23:33:28.035312 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:28.035216 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p"] Oct 29 23:33:28.039221 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:33:28.039193 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc109ab0d_5a84_45d3_bb8f_9e29a34d416a.slice/crio-997b40615993a51e71ec6921516c210a78092796add97b03382a142f5b5e1d26 WatchSource:0}: Error finding container 997b40615993a51e71ec6921516c210a78092796add97b03382a142f5b5e1d26: Status 404 returned error can't find the container with id 997b40615993a51e71ec6921516c210a78092796add97b03382a142f5b5e1d26 Oct 29 23:33:28.690211 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:28.690169 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" event={"ID":"c109ab0d-5a84-45d3-bb8f-9e29a34d416a","Type":"ContainerStarted","Data":"521991e634d099d910846f01bb698ebaf0772e65abf02a5f7a269234eccb4942"} Oct 29 23:33:28.690719 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:28.690236 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" event={"ID":"c109ab0d-5a84-45d3-bb8f-9e29a34d416a","Type":"ContainerStarted","Data":"997b40615993a51e71ec6921516c210a78092796add97b03382a142f5b5e1d26"} Oct 29 23:33:28.727304 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:28.727252 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" podStartSLOduration=1.7272327459999999 podStartE2EDuration="1.727232746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:28.723762248 +0000 UTC m=+1178.086766971" watchObservedRunningTime="2026-10-29 23:33:28.727232746 +0000 UTC m=+1178.090237469" Oct 29 23:33:29.697839 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:29.697800 6507 generic.go:356] "Generic (PLEG): container finished" podID="c109ab0d-5a84-45d3-bb8f-9e29a34d416a" containerID="521991e634d099d910846f01bb698ebaf0772e65abf02a5f7a269234eccb4942" exitCode=0 Oct 29 23:33:29.698249 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:29.697892 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" event={"ID":"c109ab0d-5a84-45d3-bb8f-9e29a34d416a","Type":"ContainerDied","Data":"521991e634d099d910846f01bb698ebaf0772e65abf02a5f7a269234eccb4942"} Oct 29 23:33:30.844896 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:30.844863 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" Oct 29 23:33:30.917701 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:30.917666 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-maas-api-openshift-ca\") pod \"c109ab0d-5a84-45d3-bb8f-9e29a34d416a\" (UID: \"c109ab0d-5a84-45d3-bb8f-9e29a34d416a\") " Oct 29 23:33:30.917905 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:30.917724 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-maas-api-xks-ca\") pod \"c109ab0d-5a84-45d3-bb8f-9e29a34d416a\" (UID: \"c109ab0d-5a84-45d3-bb8f-9e29a34d416a\") " Oct 29 23:33:30.917905 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:30.917825 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-kube-api-access-92bz2\" (UniqueName: \"kubernetes.io/projected/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-kube-api-access-92bz2\") pod \"c109ab0d-5a84-45d3-bb8f-9e29a34d416a\" (UID: \"c109ab0d-5a84-45d3-bb8f-9e29a34d416a\") " Oct 29 23:33:30.918106 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:30.918078 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-maas-api-openshift-ca" pod "c109ab0d-5a84-45d3-bb8f-9e29a34d416a" (UID: "c109ab0d-5a84-45d3-bb8f-9e29a34d416a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:30.919944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:30.919914 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-kube-api-access-92bz2" pod "c109ab0d-5a84-45d3-bb8f-9e29a34d416a" (UID: "c109ab0d-5a84-45d3-bb8f-9e29a34d416a"). InnerVolumeSpecName "kube-api-access-92bz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:30.920053 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:30.919953 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-maas-api-xks-ca" pod "c109ab0d-5a84-45d3-bb8f-9e29a34d416a" (UID: "c109ab0d-5a84-45d3-bb8f-9e29a34d416a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:33:31.019391 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:31.019296 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:33:31.019391 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:31.019334 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92bz2\" (UniqueName: \"kubernetes.io/projected/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-kube-api-access-92bz2\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:33:31.019391 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:31.019345 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c109ab0d-5a84-45d3-bb8f-9e29a34d416a-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:33:31.718394 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:31.718356 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" Oct 29 23:33:31.718394 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:31.718379 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p" event={"ID":"c109ab0d-5a84-45d3-bb8f-9e29a34d416a","Type":"ContainerDied","Data":"997b40615993a51e71ec6921516c210a78092796add97b03382a142f5b5e1d26"} Oct 29 23:33:31.718645 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:31.718484 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="997b40615993a51e71ec6921516c210a78092796add97b03382a142f5b5e1d26" Oct 29 23:33:37.031571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:37.031531 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq"] Oct 29 23:33:37.041068 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:37.041031 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28be6ca0-47wkq"] Oct 29 23:33:38.677406 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:38.677373 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:33:38.751270 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:38.751224 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cda96310-6dec-41a2-8429-1751247210ab" path="/var/lib/kubelet/pods/cda96310-6dec-41a2-8429-1751247210ab/volumes" Oct 29 23:33:39.026447 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:39.026412 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf"] Oct 29 23:33:39.034801 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:39.034766 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81beb87f-jzrjf"] Oct 29 23:33:40.751343 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:40.751300 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ec50ce-e1cb-4201-9a60-147f1bcb1db8" path="/var/lib/kubelet/pods/49ec50ce-e1cb-4201-9a60-147f1bcb1db8/volumes" Oct 29 23:33:42.813477 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:42.813423 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:33:45.354482 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:33:45.353639 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:33:48.427794 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.427753 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9"] Oct 29 23:33:48.428421 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.428398 6507 cpu_manager.go:414] "containerMap: removing container" podUID="c109ab0d-5a84-45d3-bb8f-9e29a34d416a" containerName="revoke-keys" Oct 29 23:33:48.428547 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.428423 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c109ab0d-5a84-45d3-bb8f-9e29a34d416a" containerName="revoke-keys" Oct 29 23:33:48.428547 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.428538 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="c109ab0d-5a84-45d3-bb8f-9e29a34d416a" containerName="revoke-keys" Oct 29 23:33:48.433139 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.433116 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9" Oct 29 23:33:48.445518 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.445494 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:33:48.454862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.454818 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:33:48.454982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.454854 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:33:48.455192 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.455171 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:33:48.457750 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.457720 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9"] Oct 29 23:33:48.521102 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.521073 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft7v2\" (UniqueName: \"kubernetes.io/projected/99172142-8d97-4b7b-87cb-9e1f8349633e-kube-api-access-ft7v2\") pod \"maas-api-revoke-sub-2e1f5b5f-znjl9\" (UID: \"99172142-8d97-4b7b-87cb-9e1f8349633e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9" Oct 29 23:33:48.521262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.521113 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99172142-8d97-4b7b-87cb-9e1f8349633e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-znjl9\" (UID: \"99172142-8d97-4b7b-87cb-9e1f8349633e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9" Oct 29 23:33:48.521262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.521209 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99172142-8d97-4b7b-87cb-9e1f8349633e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-znjl9\" (UID: \"99172142-8d97-4b7b-87cb-9e1f8349633e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9" Oct 29 23:33:48.622141 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.622105 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99172142-8d97-4b7b-87cb-9e1f8349633e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-znjl9\" (UID: \"99172142-8d97-4b7b-87cb-9e1f8349633e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9" Oct 29 23:33:48.622321 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.622160 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99172142-8d97-4b7b-87cb-9e1f8349633e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-znjl9\" (UID: \"99172142-8d97-4b7b-87cb-9e1f8349633e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9" Oct 29 23:33:48.622321 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.622232 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7v2\" (UniqueName: \"kubernetes.io/projected/99172142-8d97-4b7b-87cb-9e1f8349633e-kube-api-access-ft7v2\") pod \"maas-api-revoke-sub-2e1f5b5f-znjl9\" (UID: \"99172142-8d97-4b7b-87cb-9e1f8349633e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9" Oct 29 23:33:48.622918 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.622892 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99172142-8d97-4b7b-87cb-9e1f8349633e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-znjl9\" (UID: \"99172142-8d97-4b7b-87cb-9e1f8349633e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9" Oct 29 23:33:48.624512 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.624489 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99172142-8d97-4b7b-87cb-9e1f8349633e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-znjl9\" (UID: \"99172142-8d97-4b7b-87cb-9e1f8349633e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9" Oct 29 23:33:48.638748 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.638721 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft7v2\" (UniqueName: \"kubernetes.io/projected/99172142-8d97-4b7b-87cb-9e1f8349633e-kube-api-access-ft7v2\") pod \"maas-api-revoke-sub-2e1f5b5f-znjl9\" (UID: \"99172142-8d97-4b7b-87cb-9e1f8349633e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9" Oct 29 23:33:48.745916 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:48.745874 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9" Oct 29 23:33:49.026479 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:49.026372 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh"] Oct 29 23:33:49.028725 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:49.028699 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae32e799-p29hh"] Oct 29 23:33:49.043955 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:49.043932 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9"] Oct 29 23:33:49.051212 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:33:49.051185 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99172142_8d97_4b7b_87cb_9e1f8349633e.slice/crio-e65bc22e375823298372d1e428162986282c626e1809b13b53516d9f96a38a4a WatchSource:0}: Error finding container e65bc22e375823298372d1e428162986282c626e1809b13b53516d9f96a38a4a: Status 404 returned error can't find the container with id e65bc22e375823298372d1e428162986282c626e1809b13b53516d9f96a38a4a Oct 29 23:33:49.853314 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:49.853280 6507 generic.go:356] "Generic (PLEG): container finished" podID="99172142-8d97-4b7b-87cb-9e1f8349633e" containerID="d5380f2d8c08083f65761629476755517137161e334d1854b85b36c7cc8b7497" exitCode=0 Oct 29 23:33:49.853685 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:49.853318 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9" event={"ID":"99172142-8d97-4b7b-87cb-9e1f8349633e","Type":"ContainerDied","Data":"d5380f2d8c08083f65761629476755517137161e334d1854b85b36c7cc8b7497"} Oct 29 23:33:49.853685 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:49.853377 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9" event={"ID":"99172142-8d97-4b7b-87cb-9e1f8349633e","Type":"ContainerStarted","Data":"e65bc22e375823298372d1e428162986282c626e1809b13b53516d9f96a38a4a"} Oct 29 23:33:50.751950 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:50.751912 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28" path="/var/lib/kubelet/pods/b4a7ecc7-ee5f-4e7c-baa1-30758d4cbf28/volumes" Oct 29 23:33:51.004356 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:51.004290 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9" Oct 29 23:33:51.024491 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:51.024450 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9"] Oct 29 23:33:51.027616 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:51.027590 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba1c451-5b9s9"] Oct 29 23:33:51.042154 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:51.042123 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99172142-8d97-4b7b-87cb-9e1f8349633e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99172142-8d97-4b7b-87cb-9e1f8349633e-maas-api-openshift-ca\") pod \"99172142-8d97-4b7b-87cb-9e1f8349633e\" (UID: \"99172142-8d97-4b7b-87cb-9e1f8349633e\") " Oct 29 23:33:51.042325 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:51.042177 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99172142-8d97-4b7b-87cb-9e1f8349633e-kube-api-access-ft7v2\" (UniqueName: \"kubernetes.io/projected/99172142-8d97-4b7b-87cb-9e1f8349633e-kube-api-access-ft7v2\") pod \"99172142-8d97-4b7b-87cb-9e1f8349633e\" (UID: \"99172142-8d97-4b7b-87cb-9e1f8349633e\") " Oct 29 23:33:51.042920 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:51.042872 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99172142-8d97-4b7b-87cb-9e1f8349633e-maas-api-openshift-ca" pod "99172142-8d97-4b7b-87cb-9e1f8349633e" (UID: "99172142-8d97-4b7b-87cb-9e1f8349633e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:51.043069 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:51.043042 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99172142-8d97-4b7b-87cb-9e1f8349633e-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:33:51.053212 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:51.053165 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99172142-8d97-4b7b-87cb-9e1f8349633e-kube-api-access-ft7v2" pod "99172142-8d97-4b7b-87cb-9e1f8349633e" (UID: "99172142-8d97-4b7b-87cb-9e1f8349633e"). InnerVolumeSpecName "kube-api-access-ft7v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:51.144026 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:51.143994 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99172142-8d97-4b7b-87cb-9e1f8349633e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99172142-8d97-4b7b-87cb-9e1f8349633e-maas-api-xks-ca\") pod \"99172142-8d97-4b7b-87cb-9e1f8349633e\" (UID: \"99172142-8d97-4b7b-87cb-9e1f8349633e\") " Oct 29 23:33:51.144297 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:51.144283 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft7v2\" (UniqueName: \"kubernetes.io/projected/99172142-8d97-4b7b-87cb-9e1f8349633e-kube-api-access-ft7v2\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:33:51.146120 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:51.146084 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99172142-8d97-4b7b-87cb-9e1f8349633e-maas-api-xks-ca" pod "99172142-8d97-4b7b-87cb-9e1f8349633e" (UID: "99172142-8d97-4b7b-87cb-9e1f8349633e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:33:51.245428 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:51.245386 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99172142-8d97-4b7b-87cb-9e1f8349633e-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:33:51.865410 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:51.865378 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9" Oct 29 23:33:51.865644 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:51.865413 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9" event={"ID":"99172142-8d97-4b7b-87cb-9e1f8349633e","Type":"ContainerDied","Data":"e65bc22e375823298372d1e428162986282c626e1809b13b53516d9f96a38a4a"} Oct 29 23:33:51.865644 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:51.865482 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e65bc22e375823298372d1e428162986282c626e1809b13b53516d9f96a38a4a" Oct 29 23:33:52.751115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:52.751082 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c944a028-57d8-4711-a4ba-f581a6148bd8" path="/var/lib/kubelet/pods/c944a028-57d8-4711-a4ba-f581a6148bd8/volumes" Oct 29 23:33:52.773572 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:52.773541 6507 scope.go:122] "RemoveContainer" containerID="7153e2f79f052ff3e47104c5e10ef47b4edd211c982c74be87f95d0a36983d2f" Oct 29 23:33:52.782344 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:52.782324 6507 scope.go:122] "RemoveContainer" containerID="794be87a9b78f990e7fc40197ef2874591420656b3902017811406e6b5fdb378" Oct 29 23:33:52.790624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:52.790603 6507 scope.go:122] "RemoveContainer" containerID="f9693e0ef38e60c797c61b7c840199c9e6ade5a290bacfd9d7aaeee5906d5ee3" Oct 29 23:33:59.654150 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.654115 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq"] Oct 29 23:33:59.654614 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.654573 6507 cpu_manager.go:414] "containerMap: removing container" podUID="99172142-8d97-4b7b-87cb-9e1f8349633e" containerName="revoke-keys" Oct 29 23:33:59.654614 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.654587 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99172142-8d97-4b7b-87cb-9e1f8349633e" containerName="revoke-keys" Oct 29 23:33:59.654691 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.654648 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="99172142-8d97-4b7b-87cb-9e1f8349633e" containerName="revoke-keys" Oct 29 23:33:59.659041 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.659025 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" Oct 29 23:33:59.662363 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.662296 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:33:59.671648 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.671624 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:33:59.671648 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.671631 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:33:59.671789 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.671693 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:33:59.671828 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.671806 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:33:59.683433 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.683402 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq"] Oct 29 23:33:59.716425 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.716381 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2cqf\" (UniqueName: \"kubernetes.io/projected/09a4dc6f-8533-4917-8da8-4902ebec72d1-kube-api-access-n2cqf\") pod \"maas-api-revoke-sub-ec843018-bxnvq\" (UID: \"09a4dc6f-8533-4917-8da8-4902ebec72d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" Oct 29 23:33:59.716600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.716554 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a4dc6f-8533-4917-8da8-4902ebec72d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bxnvq\" (UID: \"09a4dc6f-8533-4917-8da8-4902ebec72d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" Oct 29 23:33:59.716703 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.716681 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a4dc6f-8533-4917-8da8-4902ebec72d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bxnvq\" (UID: \"09a4dc6f-8533-4917-8da8-4902ebec72d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" Oct 29 23:33:59.817216 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.817177 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a4dc6f-8533-4917-8da8-4902ebec72d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bxnvq\" (UID: \"09a4dc6f-8533-4917-8da8-4902ebec72d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" Oct 29 23:33:59.817435 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.817241 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2cqf\" (UniqueName: \"kubernetes.io/projected/09a4dc6f-8533-4917-8da8-4902ebec72d1-kube-api-access-n2cqf\") pod \"maas-api-revoke-sub-ec843018-bxnvq\" (UID: \"09a4dc6f-8533-4917-8da8-4902ebec72d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" Oct 29 23:33:59.817435 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.817293 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a4dc6f-8533-4917-8da8-4902ebec72d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bxnvq\" (UID: \"09a4dc6f-8533-4917-8da8-4902ebec72d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" Oct 29 23:33:59.818077 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.818057 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a4dc6f-8533-4917-8da8-4902ebec72d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bxnvq\" (UID: \"09a4dc6f-8533-4917-8da8-4902ebec72d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" Oct 29 23:33:59.819512 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.819494 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a4dc6f-8533-4917-8da8-4902ebec72d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bxnvq\" (UID: \"09a4dc6f-8533-4917-8da8-4902ebec72d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" Oct 29 23:33:59.833849 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.833822 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2cqf\" (UniqueName: \"kubernetes.io/projected/09a4dc6f-8533-4917-8da8-4902ebec72d1-kube-api-access-n2cqf\") pod \"maas-api-revoke-sub-ec843018-bxnvq\" (UID: \"09a4dc6f-8533-4917-8da8-4902ebec72d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" Oct 29 23:33:59.972429 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:33:59.972395 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" Oct 29 23:34:00.063235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:00.031629 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w"] Oct 29 23:34:00.063235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:00.035838 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50a539e3-dz55w"] Oct 29 23:34:00.271571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:00.271489 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq"] Oct 29 23:34:00.275075 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:34:00.275049 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a4dc6f_8533_4917_8da8_4902ebec72d1.slice/crio-d5f005a5c893d555bf1ab6795e7fcdaaccd6b526089f2b927828832d2256e8fe WatchSource:0}: Error finding container d5f005a5c893d555bf1ab6795e7fcdaaccd6b526089f2b927828832d2256e8fe: Status 404 returned error can't find the container with id d5f005a5c893d555bf1ab6795e7fcdaaccd6b526089f2b927828832d2256e8fe Oct 29 23:34:00.751338 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:00.751302 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10904812-7716-478e-95c0-796a65c4d269" path="/var/lib/kubelet/pods/10904812-7716-478e-95c0-796a65c4d269/volumes" Oct 29 23:34:00.915762 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:00.915679 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" event={"ID":"09a4dc6f-8533-4917-8da8-4902ebec72d1","Type":"ContainerStarted","Data":"851d5ce1f272f113fcbb6307b3e514a4d998f80cdf4e91b39dc62e684f28ba85"} Oct 29 23:34:00.915762 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:00.915736 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" event={"ID":"09a4dc6f-8533-4917-8da8-4902ebec72d1","Type":"ContainerStarted","Data":"d5f005a5c893d555bf1ab6795e7fcdaaccd6b526089f2b927828832d2256e8fe"} Oct 29 23:34:00.950629 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:00.950566 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" podStartSLOduration=1.950550626 podStartE2EDuration="1.950550626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:34:00.948873154 +0000 UTC m=+1210.311877877" watchObservedRunningTime="2026-10-29 23:34:00.950550626 +0000 UTC m=+1210.313555349" Oct 29 23:34:01.923708 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:01.923671 6507 generic.go:356] "Generic (PLEG): container finished" podID="09a4dc6f-8533-4917-8da8-4902ebec72d1" containerID="851d5ce1f272f113fcbb6307b3e514a4d998f80cdf4e91b39dc62e684f28ba85" exitCode=0 Oct 29 23:34:01.924242 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:01.923737 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" event={"ID":"09a4dc6f-8533-4917-8da8-4902ebec72d1","Type":"ContainerDied","Data":"851d5ce1f272f113fcbb6307b3e514a4d998f80cdf4e91b39dc62e684f28ba85"} Oct 29 23:34:03.067897 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:03.067869 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" Oct 29 23:34:03.151220 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:03.151183 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a4dc6f-8533-4917-8da8-4902ebec72d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a4dc6f-8533-4917-8da8-4902ebec72d1-maas-api-openshift-ca\") pod \"09a4dc6f-8533-4917-8da8-4902ebec72d1\" (UID: \"09a4dc6f-8533-4917-8da8-4902ebec72d1\") " Oct 29 23:34:03.151421 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:03.151309 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09a4dc6f-8533-4917-8da8-4902ebec72d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a4dc6f-8533-4917-8da8-4902ebec72d1-maas-api-xks-ca\") pod \"09a4dc6f-8533-4917-8da8-4902ebec72d1\" (UID: \"09a4dc6f-8533-4917-8da8-4902ebec72d1\") " Oct 29 23:34:03.151421 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:03.151355 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a4dc6f-8533-4917-8da8-4902ebec72d1-kube-api-access-n2cqf\" (UniqueName: \"kubernetes.io/projected/09a4dc6f-8533-4917-8da8-4902ebec72d1-kube-api-access-n2cqf\") pod \"09a4dc6f-8533-4917-8da8-4902ebec72d1\" (UID: \"09a4dc6f-8533-4917-8da8-4902ebec72d1\") " Oct 29 23:34:03.151663 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:03.151639 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a4dc6f-8533-4917-8da8-4902ebec72d1-maas-api-openshift-ca" pod "09a4dc6f-8533-4917-8da8-4902ebec72d1" (UID: "09a4dc6f-8533-4917-8da8-4902ebec72d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:34:03.153500 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:03.153468 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a4dc6f-8533-4917-8da8-4902ebec72d1-maas-api-xks-ca" pod "09a4dc6f-8533-4917-8da8-4902ebec72d1" (UID: "09a4dc6f-8533-4917-8da8-4902ebec72d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:34:03.153500 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:03.153488 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a4dc6f-8533-4917-8da8-4902ebec72d1-kube-api-access-n2cqf" pod "09a4dc6f-8533-4917-8da8-4902ebec72d1" (UID: "09a4dc6f-8533-4917-8da8-4902ebec72d1"). InnerVolumeSpecName "kube-api-access-n2cqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:34:03.252285 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:03.252245 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a4dc6f-8533-4917-8da8-4902ebec72d1-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:34:03.252285 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:03.252278 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a4dc6f-8533-4917-8da8-4902ebec72d1-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:34:03.252285 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:03.252288 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2cqf\" (UniqueName: \"kubernetes.io/projected/09a4dc6f-8533-4917-8da8-4902ebec72d1-kube-api-access-n2cqf\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:34:03.937179 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:03.937151 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" Oct 29 23:34:03.937359 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:03.937145 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq" event={"ID":"09a4dc6f-8533-4917-8da8-4902ebec72d1","Type":"ContainerDied","Data":"d5f005a5c893d555bf1ab6795e7fcdaaccd6b526089f2b927828832d2256e8fe"} Oct 29 23:34:03.937359 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:03.937241 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5f005a5c893d555bf1ab6795e7fcdaaccd6b526089f2b927828832d2256e8fe" Oct 29 23:34:06.025029 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:06.024992 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn"] Oct 29 23:34:06.034077 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:06.034050 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd375e44-m78nn"] Oct 29 23:34:06.752327 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:06.752291 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e8e6875-435c-4ae7-830e-bbabe94589c0" path="/var/lib/kubelet/pods/4e8e6875-435c-4ae7-830e-bbabe94589c0/volumes" Oct 29 23:34:15.201900 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:15.201843 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:34:15.222717 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:34:15.222674 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:34:17.772606 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:17.772565 6507 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-bsqhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:34:17.773522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:17.773186 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-bsqhf" podUID="7084c3a6-6454-456d-a6ad-06f58bf3e4e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:34:27.032476 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.032433 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7"] Oct 29 23:34:27.041566 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.041535 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-935b71b4-zthz7"] Oct 29 23:34:27.597014 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.596978 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td"] Oct 29 23:34:27.597444 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.597429 6507 cpu_manager.go:414] "containerMap: removing container" podUID="09a4dc6f-8533-4917-8da8-4902ebec72d1" containerName="revoke-keys" Oct 29 23:34:27.597521 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.597445 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a4dc6f-8533-4917-8da8-4902ebec72d1" containerName="revoke-keys" Oct 29 23:34:27.597561 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.597536 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a4dc6f-8533-4917-8da8-4902ebec72d1" containerName="revoke-keys" Oct 29 23:34:27.600981 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.600961 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td" Oct 29 23:34:27.622386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.622356 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:34:27.622593 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.622385 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:34:27.622593 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.622427 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:34:27.622725 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.622645 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:34:27.625181 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.625159 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td"] Oct 29 23:34:27.745854 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.745808 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd5k2\" (UniqueName: \"kubernetes.io/projected/a0de3c32-2943-469d-ac61-b6377f30c220-kube-api-access-cd5k2\") pod \"maas-api-revoke-sub-046e1343-xz2td\" (UID: \"a0de3c32-2943-469d-ac61-b6377f30c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td" Oct 29 23:34:27.746064 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.745958 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0de3c32-2943-469d-ac61-b6377f30c220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xz2td\" (UID: \"a0de3c32-2943-469d-ac61-b6377f30c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td" Oct 29 23:34:27.746142 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.746069 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0de3c32-2943-469d-ac61-b6377f30c220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xz2td\" (UID: \"a0de3c32-2943-469d-ac61-b6377f30c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td" Oct 29 23:34:27.846908 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.846872 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0de3c32-2943-469d-ac61-b6377f30c220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xz2td\" (UID: \"a0de3c32-2943-469d-ac61-b6377f30c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td" Oct 29 23:34:27.847139 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.846937 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0de3c32-2943-469d-ac61-b6377f30c220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xz2td\" (UID: \"a0de3c32-2943-469d-ac61-b6377f30c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td" Oct 29 23:34:27.847139 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.846994 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd5k2\" (UniqueName: \"kubernetes.io/projected/a0de3c32-2943-469d-ac61-b6377f30c220-kube-api-access-cd5k2\") pod \"maas-api-revoke-sub-046e1343-xz2td\" (UID: \"a0de3c32-2943-469d-ac61-b6377f30c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td" Oct 29 23:34:27.847706 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.847685 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0de3c32-2943-469d-ac61-b6377f30c220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xz2td\" (UID: \"a0de3c32-2943-469d-ac61-b6377f30c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td" Oct 29 23:34:27.849400 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.849379 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0de3c32-2943-469d-ac61-b6377f30c220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xz2td\" (UID: \"a0de3c32-2943-469d-ac61-b6377f30c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td" Oct 29 23:34:27.864708 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.864672 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd5k2\" (UniqueName: \"kubernetes.io/projected/a0de3c32-2943-469d-ac61-b6377f30c220-kube-api-access-cd5k2\") pod \"maas-api-revoke-sub-046e1343-xz2td\" (UID: \"a0de3c32-2943-469d-ac61-b6377f30c220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td" Oct 29 23:34:27.914436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:27.914401 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td" Oct 29 23:34:28.211538 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:28.211504 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td"] Oct 29 23:34:28.215342 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:34:28.215310 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0de3c32_2943_469d_ac61_b6377f30c220.slice/crio-eedd786d94fafcda47e93fb4feb69097f206a1c1663b2cfa4dec7a89e8f160d3 WatchSource:0}: Error finding container eedd786d94fafcda47e93fb4feb69097f206a1c1663b2cfa4dec7a89e8f160d3: Status 404 returned error can't find the container with id eedd786d94fafcda47e93fb4feb69097f206a1c1663b2cfa4dec7a89e8f160d3 Oct 29 23:34:28.751731 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:28.751696 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efed76ac-e277-4ad2-bf5d-7924ad9bd181" path="/var/lib/kubelet/pods/efed76ac-e277-4ad2-bf5d-7924ad9bd181/volumes" Oct 29 23:34:29.135202 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:29.135166 6507 generic.go:356] "Generic (PLEG): container finished" podID="a0de3c32-2943-469d-ac61-b6377f30c220" containerID="a007326d8e904a97b20b826ff2c86460d9da9e2ccda1e7c75ac6cb990cd8fad6" exitCode=0 Oct 29 23:34:29.135417 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:29.135253 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td" event={"ID":"a0de3c32-2943-469d-ac61-b6377f30c220","Type":"ContainerDied","Data":"a007326d8e904a97b20b826ff2c86460d9da9e2ccda1e7c75ac6cb990cd8fad6"} Oct 29 23:34:29.135417 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:29.135318 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td" event={"ID":"a0de3c32-2943-469d-ac61-b6377f30c220","Type":"ContainerStarted","Data":"eedd786d94fafcda47e93fb4feb69097f206a1c1663b2cfa4dec7a89e8f160d3"} Oct 29 23:34:30.275736 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:30.275712 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td" Oct 29 23:34:30.473789 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:30.473744 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0de3c32-2943-469d-ac61-b6377f30c220-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0de3c32-2943-469d-ac61-b6377f30c220-maas-api-xks-ca\") pod \"a0de3c32-2943-469d-ac61-b6377f30c220\" (UID: \"a0de3c32-2943-469d-ac61-b6377f30c220\") " Oct 29 23:34:30.473986 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:30.473864 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0de3c32-2943-469d-ac61-b6377f30c220-kube-api-access-cd5k2\" (UniqueName: \"kubernetes.io/projected/a0de3c32-2943-469d-ac61-b6377f30c220-kube-api-access-cd5k2\") pod \"a0de3c32-2943-469d-ac61-b6377f30c220\" (UID: \"a0de3c32-2943-469d-ac61-b6377f30c220\") " Oct 29 23:34:30.473986 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:30.473915 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0de3c32-2943-469d-ac61-b6377f30c220-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0de3c32-2943-469d-ac61-b6377f30c220-maas-api-openshift-ca\") pod \"a0de3c32-2943-469d-ac61-b6377f30c220\" (UID: \"a0de3c32-2943-469d-ac61-b6377f30c220\") " Oct 29 23:34:30.474284 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:30.474257 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0de3c32-2943-469d-ac61-b6377f30c220-maas-api-openshift-ca" pod "a0de3c32-2943-469d-ac61-b6377f30c220" (UID: "a0de3c32-2943-469d-ac61-b6377f30c220"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:34:30.475869 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:30.475848 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0de3c32-2943-469d-ac61-b6377f30c220-maas-api-xks-ca" pod "a0de3c32-2943-469d-ac61-b6377f30c220" (UID: "a0de3c32-2943-469d-ac61-b6377f30c220"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:34:30.475951 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:30.475898 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0de3c32-2943-469d-ac61-b6377f30c220-kube-api-access-cd5k2" pod "a0de3c32-2943-469d-ac61-b6377f30c220" (UID: "a0de3c32-2943-469d-ac61-b6377f30c220"). InnerVolumeSpecName "kube-api-access-cd5k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:34:30.574693 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:30.574656 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd5k2\" (UniqueName: \"kubernetes.io/projected/a0de3c32-2943-469d-ac61-b6377f30c220-kube-api-access-cd5k2\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:34:30.574693 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:30.574687 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0de3c32-2943-469d-ac61-b6377f30c220-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:34:30.574693 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:30.574708 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0de3c32-2943-469d-ac61-b6377f30c220-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:34:31.147532 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:31.147425 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td" Oct 29 23:34:31.147532 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:31.147453 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td" event={"ID":"a0de3c32-2943-469d-ac61-b6377f30c220","Type":"ContainerDied","Data":"eedd786d94fafcda47e93fb4feb69097f206a1c1663b2cfa4dec7a89e8f160d3"} Oct 29 23:34:31.147532 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:31.147524 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eedd786d94fafcda47e93fb4feb69097f206a1c1663b2cfa4dec7a89e8f160d3" Oct 29 23:34:37.938520 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:37.938483 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn"] Oct 29 23:34:37.939004 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:37.938892 6507 cpu_manager.go:414] "containerMap: removing container" podUID="a0de3c32-2943-469d-ac61-b6377f30c220" containerName="revoke-keys" Oct 29 23:34:37.939004 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:37.938904 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0de3c32-2943-469d-ac61-b6377f30c220" containerName="revoke-keys" Oct 29 23:34:37.939004 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:37.938987 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0de3c32-2943-469d-ac61-b6377f30c220" containerName="revoke-keys" Oct 29 23:34:37.942189 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:37.942174 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" Oct 29 23:34:37.956687 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:37.956662 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:34:37.956827 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:37.956707 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:34:37.956827 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:37.956704 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:34:37.956938 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:37.956831 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:34:37.959741 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:37.959716 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn"] Oct 29 23:34:38.042353 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:38.042317 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4482j\" (UniqueName: \"kubernetes.io/projected/847c16f6-eacf-4f57-87f8-9dbaddbcf884-kube-api-access-4482j\") pod \"maas-api-revoke-sub-d3fba056-5zltn\" (UID: \"847c16f6-eacf-4f57-87f8-9dbaddbcf884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" Oct 29 23:34:38.042578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:38.042359 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/847c16f6-eacf-4f57-87f8-9dbaddbcf884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5zltn\" (UID: \"847c16f6-eacf-4f57-87f8-9dbaddbcf884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" Oct 29 23:34:38.042659 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:38.042571 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/847c16f6-eacf-4f57-87f8-9dbaddbcf884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5zltn\" (UID: \"847c16f6-eacf-4f57-87f8-9dbaddbcf884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" Oct 29 23:34:38.143314 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:38.143269 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/847c16f6-eacf-4f57-87f8-9dbaddbcf884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5zltn\" (UID: \"847c16f6-eacf-4f57-87f8-9dbaddbcf884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" Oct 29 23:34:38.143528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:38.143333 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4482j\" (UniqueName: \"kubernetes.io/projected/847c16f6-eacf-4f57-87f8-9dbaddbcf884-kube-api-access-4482j\") pod \"maas-api-revoke-sub-d3fba056-5zltn\" (UID: \"847c16f6-eacf-4f57-87f8-9dbaddbcf884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" Oct 29 23:34:38.143528 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:38.143372 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/847c16f6-eacf-4f57-87f8-9dbaddbcf884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5zltn\" (UID: \"847c16f6-eacf-4f57-87f8-9dbaddbcf884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" Oct 29 23:34:38.143988 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:38.143966 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/847c16f6-eacf-4f57-87f8-9dbaddbcf884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5zltn\" (UID: \"847c16f6-eacf-4f57-87f8-9dbaddbcf884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" Oct 29 23:34:38.145792 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:38.145773 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/847c16f6-eacf-4f57-87f8-9dbaddbcf884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5zltn\" (UID: \"847c16f6-eacf-4f57-87f8-9dbaddbcf884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" Oct 29 23:34:38.159741 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:38.159715 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4482j\" (UniqueName: \"kubernetes.io/projected/847c16f6-eacf-4f57-87f8-9dbaddbcf884-kube-api-access-4482j\") pod \"maas-api-revoke-sub-d3fba056-5zltn\" (UID: \"847c16f6-eacf-4f57-87f8-9dbaddbcf884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" Oct 29 23:34:38.255849 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:38.255766 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" Oct 29 23:34:38.553075 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:38.552992 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn"] Oct 29 23:34:38.557213 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:34:38.557182 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847c16f6_eacf_4f57_87f8_9dbaddbcf884.slice/crio-3ffeed899dcbdd2cda79e54bfeb193e961f0d25478a13ee5c21c3e0f79e194cd WatchSource:0}: Error finding container 3ffeed899dcbdd2cda79e54bfeb193e961f0d25478a13ee5c21c3e0f79e194cd: Status 404 returned error can't find the container with id 3ffeed899dcbdd2cda79e54bfeb193e961f0d25478a13ee5c21c3e0f79e194cd Oct 29 23:34:38.704813 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:38.704779 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:34:39.197306 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:39.197260 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" event={"ID":"847c16f6-eacf-4f57-87f8-9dbaddbcf884","Type":"ContainerStarted","Data":"22d6821168d052fd409abab6659a4db557dc4ff1c66e76bcb35ac51d4405c7f9"} Oct 29 23:34:39.197840 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:39.197324 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" event={"ID":"847c16f6-eacf-4f57-87f8-9dbaddbcf884","Type":"ContainerStarted","Data":"3ffeed899dcbdd2cda79e54bfeb193e961f0d25478a13ee5c21c3e0f79e194cd"} Oct 29 23:34:39.229306 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:39.229239 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" podStartSLOduration=2.22921892 podStartE2EDuration="2.22921892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:34:39.228810428 +0000 UTC m=+1248.591815196" watchObservedRunningTime="2026-10-29 23:34:39.22921892 +0000 UTC m=+1248.592223641" Oct 29 23:34:40.226553 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:40.226506 6507 generic.go:356] "Generic (PLEG): container finished" podID="847c16f6-eacf-4f57-87f8-9dbaddbcf884" containerID="22d6821168d052fd409abab6659a4db557dc4ff1c66e76bcb35ac51d4405c7f9" exitCode=0 Oct 29 23:34:40.227177 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:40.227129 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" event={"ID":"847c16f6-eacf-4f57-87f8-9dbaddbcf884","Type":"ContainerDied","Data":"22d6821168d052fd409abab6659a4db557dc4ff1c66e76bcb35ac51d4405c7f9"} Oct 29 23:34:41.738988 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:41.738626 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" Oct 29 23:34:41.837248 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:41.837139 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847c16f6-eacf-4f57-87f8-9dbaddbcf884-kube-api-access-4482j\" (UniqueName: \"kubernetes.io/projected/847c16f6-eacf-4f57-87f8-9dbaddbcf884-kube-api-access-4482j\") pod \"847c16f6-eacf-4f57-87f8-9dbaddbcf884\" (UID: \"847c16f6-eacf-4f57-87f8-9dbaddbcf884\") " Oct 29 23:34:41.837524 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:41.837499 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/847c16f6-eacf-4f57-87f8-9dbaddbcf884-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/847c16f6-eacf-4f57-87f8-9dbaddbcf884-maas-api-openshift-ca\") pod \"847c16f6-eacf-4f57-87f8-9dbaddbcf884\" (UID: \"847c16f6-eacf-4f57-87f8-9dbaddbcf884\") " Oct 29 23:34:41.837727 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:41.837708 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/847c16f6-eacf-4f57-87f8-9dbaddbcf884-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/847c16f6-eacf-4f57-87f8-9dbaddbcf884-maas-api-xks-ca\") pod \"847c16f6-eacf-4f57-87f8-9dbaddbcf884\" (UID: \"847c16f6-eacf-4f57-87f8-9dbaddbcf884\") " Oct 29 23:34:41.839564 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:41.839538 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847c16f6-eacf-4f57-87f8-9dbaddbcf884-maas-api-openshift-ca" pod "847c16f6-eacf-4f57-87f8-9dbaddbcf884" (UID: "847c16f6-eacf-4f57-87f8-9dbaddbcf884"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:34:41.855972 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:41.849162 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847c16f6-eacf-4f57-87f8-9dbaddbcf884-maas-api-xks-ca" pod "847c16f6-eacf-4f57-87f8-9dbaddbcf884" (UID: "847c16f6-eacf-4f57-87f8-9dbaddbcf884"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:34:41.855972 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:41.854914 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847c16f6-eacf-4f57-87f8-9dbaddbcf884-kube-api-access-4482j" pod "847c16f6-eacf-4f57-87f8-9dbaddbcf884" (UID: "847c16f6-eacf-4f57-87f8-9dbaddbcf884"). InnerVolumeSpecName "kube-api-access-4482j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:34:41.939104 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:41.939065 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4482j\" (UniqueName: \"kubernetes.io/projected/847c16f6-eacf-4f57-87f8-9dbaddbcf884-kube-api-access-4482j\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:34:41.939104 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:41.939100 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/847c16f6-eacf-4f57-87f8-9dbaddbcf884-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:34:41.939104 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:41.939111 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/847c16f6-eacf-4f57-87f8-9dbaddbcf884-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:34:42.248313 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:42.248280 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" Oct 29 23:34:42.248313 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:42.248298 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn" event={"ID":"847c16f6-eacf-4f57-87f8-9dbaddbcf884","Type":"ContainerDied","Data":"3ffeed899dcbdd2cda79e54bfeb193e961f0d25478a13ee5c21c3e0f79e194cd"} Oct 29 23:34:42.248621 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:42.248364 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ffeed899dcbdd2cda79e54bfeb193e961f0d25478a13ee5c21c3e0f79e194cd" Oct 29 23:34:44.025199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:44.025134 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc"] Oct 29 23:34:44.034074 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:44.034034 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01e07b87-c54jc"] Oct 29 23:34:44.751086 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:44.751041 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e69bec15-7520-4331-a8af-bf1cefd3fd1e" path="/var/lib/kubelet/pods/e69bec15-7520-4331-a8af-bf1cefd3fd1e/volumes" Oct 29 23:34:45.223812 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:34:45.223776 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:34:46.026499 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:46.026443 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6"] Oct 29 23:34:46.035659 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:46.035622 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e99cfdf5-lg6l6"] Oct 29 23:34:46.751619 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:46.751582 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60bf6b38-c0c8-4e8b-bd72-6876edf4dae3" path="/var/lib/kubelet/pods/60bf6b38-c0c8-4e8b-bd72-6876edf4dae3/volumes" Oct 29 23:34:49.089970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:49.089929 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:34:52.823606 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:52.823571 6507 scope.go:122] "RemoveContainer" containerID="ab3e5fe6c1c77678dc01b75efb38d8fd1f0a066fdc15801f71cc2c49d9736734" Oct 29 23:34:52.832645 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:52.832620 6507 scope.go:122] "RemoveContainer" containerID="0ad492ed4611013494b4a903fc6ebf68eec9e849b1cac8246eb5ef50bbba2ebd" Oct 29 23:34:52.840702 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:52.840675 6507 scope.go:122] "RemoveContainer" containerID="13b102c610d2c8922fa00579135284f30ba19652e818b6b8b46604f42bebbe6d" Oct 29 23:34:52.848787 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:52.848766 6507 scope.go:122] "RemoveContainer" containerID="0c17c5edacd5fd378a2fa8b16019dc4779f3990399a537e77b16b1807277199c" Oct 29 23:34:52.856771 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:52.856752 6507 scope.go:122] "RemoveContainer" containerID="0c67e97d3ff7d369181c44001e236c3a2756a6dcaa02b231ce84fa9965e7f0d9" Oct 29 23:34:53.032146 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:53.032093 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq"] Oct 29 23:34:53.034429 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:53.034401 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50d79429-vfqwq"] Oct 29 23:34:53.075742 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:53.075662 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:34:54.774494 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:54.763370 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21406741-12e6-4996-95b1-23be60d44ee1" path="/var/lib/kubelet/pods/21406741-12e6-4996-95b1-23be60d44ee1/volumes" Oct 29 23:34:59.430050 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.430015 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24"] Oct 29 23:34:59.430604 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.430562 6507 cpu_manager.go:414] "containerMap: removing container" podUID="847c16f6-eacf-4f57-87f8-9dbaddbcf884" containerName="revoke-keys" Oct 29 23:34:59.430604 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.430578 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847c16f6-eacf-4f57-87f8-9dbaddbcf884" containerName="revoke-keys" Oct 29 23:34:59.430722 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.430686 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="847c16f6-eacf-4f57-87f8-9dbaddbcf884" containerName="revoke-keys" Oct 29 23:34:59.438148 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.438118 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" Oct 29 23:34:59.451043 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.451022 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:34:59.460724 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.460693 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:34:59.460724 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.460682 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:34:59.460920 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.460782 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:34:59.464370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.464344 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24"] Oct 29 23:34:59.577911 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.577867 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqsjd\" (UniqueName: \"kubernetes.io/projected/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-kube-api-access-mqsjd\") pod \"maas-api-revoke-sub-ae594979-9zb24\" (UID: \"e7c73b24-9b5d-4883-9873-8eb1be6a6b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" Oct 29 23:34:59.578116 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.578040 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-9zb24\" (UID: \"e7c73b24-9b5d-4883-9873-8eb1be6a6b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" Oct 29 23:34:59.578116 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.578097 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-9zb24\" (UID: \"e7c73b24-9b5d-4883-9873-8eb1be6a6b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" Oct 29 23:34:59.679714 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.679676 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-9zb24\" (UID: \"e7c73b24-9b5d-4883-9873-8eb1be6a6b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" Oct 29 23:34:59.679714 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.679728 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-9zb24\" (UID: \"e7c73b24-9b5d-4883-9873-8eb1be6a6b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" Oct 29 23:34:59.679940 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.679797 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqsjd\" (UniqueName: \"kubernetes.io/projected/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-kube-api-access-mqsjd\") pod \"maas-api-revoke-sub-ae594979-9zb24\" (UID: \"e7c73b24-9b5d-4883-9873-8eb1be6a6b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" Oct 29 23:34:59.680529 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.680443 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-9zb24\" (UID: \"e7c73b24-9b5d-4883-9873-8eb1be6a6b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" Oct 29 23:34:59.682173 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.682143 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-9zb24\" (UID: \"e7c73b24-9b5d-4883-9873-8eb1be6a6b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" Oct 29 23:34:59.696541 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.696512 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqsjd\" (UniqueName: \"kubernetes.io/projected/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-kube-api-access-mqsjd\") pod \"maas-api-revoke-sub-ae594979-9zb24\" (UID: \"e7c73b24-9b5d-4883-9873-8eb1be6a6b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" Oct 29 23:34:59.751171 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:34:59.751126 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" Oct 29 23:35:00.023678 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:00.023615 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h"] Oct 29 23:35:00.025777 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:00.025753 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ff301fd-wrx5h"] Oct 29 23:35:00.062773 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:00.062739 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24"] Oct 29 23:35:00.066430 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:35:00.066393 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c73b24_9b5d_4883_9873_8eb1be6a6b68.slice/crio-e1cb966cb3dcab126979b0c2fe5f76723800f13e70e01e2f86d6d6c2354d81b1 WatchSource:0}: Error finding container e1cb966cb3dcab126979b0c2fe5f76723800f13e70e01e2f86d6d6c2354d81b1: Status 404 returned error can't find the container with id e1cb966cb3dcab126979b0c2fe5f76723800f13e70e01e2f86d6d6c2354d81b1 Oct 29 23:35:00.386561 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:00.386449 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" event={"ID":"e7c73b24-9b5d-4883-9873-8eb1be6a6b68","Type":"ContainerStarted","Data":"8bc0ef02dce3b1968c61ed3941f6c2ad0a0b0a8b1a6839fcc229bc3b8b3371dd"} Oct 29 23:35:00.386561 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:00.386529 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" event={"ID":"e7c73b24-9b5d-4883-9873-8eb1be6a6b68","Type":"ContainerStarted","Data":"e1cb966cb3dcab126979b0c2fe5f76723800f13e70e01e2f86d6d6c2354d81b1"} Oct 29 23:35:00.420638 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:00.420577 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" podStartSLOduration=1.420557708 podStartE2EDuration="1.420557708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:35:00.418801288 +0000 UTC m=+1269.781806011" watchObservedRunningTime="2026-10-29 23:35:00.420557708 +0000 UTC m=+1269.783562432" Oct 29 23:35:00.752026 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:00.751997 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87da7dbd-9b2c-4cc4-98c9-1a302d52ce68" path="/var/lib/kubelet/pods/87da7dbd-9b2c-4cc4-98c9-1a302d52ce68/volumes" Oct 29 23:35:01.393081 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:01.393049 6507 generic.go:356] "Generic (PLEG): container finished" podID="e7c73b24-9b5d-4883-9873-8eb1be6a6b68" containerID="8bc0ef02dce3b1968c61ed3941f6c2ad0a0b0a8b1a6839fcc229bc3b8b3371dd" exitCode=0 Oct 29 23:35:01.395174 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:01.393131 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" event={"ID":"e7c73b24-9b5d-4883-9873-8eb1be6a6b68","Type":"ContainerDied","Data":"8bc0ef02dce3b1968c61ed3941f6c2ad0a0b0a8b1a6839fcc229bc3b8b3371dd"} Oct 29 23:35:02.539815 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:02.539787 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" Oct 29 23:35:02.710271 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:02.710239 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-maas-api-xks-ca\") pod \"e7c73b24-9b5d-4883-9873-8eb1be6a6b68\" (UID: \"e7c73b24-9b5d-4883-9873-8eb1be6a6b68\") " Oct 29 23:35:02.710436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:02.710301 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-maas-api-openshift-ca\") pod \"e7c73b24-9b5d-4883-9873-8eb1be6a6b68\" (UID: \"e7c73b24-9b5d-4883-9873-8eb1be6a6b68\") " Oct 29 23:35:02.710436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:02.710324 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-kube-api-access-mqsjd\" (UniqueName: \"kubernetes.io/projected/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-kube-api-access-mqsjd\") pod \"e7c73b24-9b5d-4883-9873-8eb1be6a6b68\" (UID: \"e7c73b24-9b5d-4883-9873-8eb1be6a6b68\") " Oct 29 23:35:02.710734 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:02.710707 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-maas-api-openshift-ca" pod "e7c73b24-9b5d-4883-9873-8eb1be6a6b68" (UID: "e7c73b24-9b5d-4883-9873-8eb1be6a6b68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:35:02.712506 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:02.712480 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-maas-api-xks-ca" pod "e7c73b24-9b5d-4883-9873-8eb1be6a6b68" (UID: "e7c73b24-9b5d-4883-9873-8eb1be6a6b68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:35:02.712624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:02.712526 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-kube-api-access-mqsjd" pod "e7c73b24-9b5d-4883-9873-8eb1be6a6b68" (UID: "e7c73b24-9b5d-4883-9873-8eb1be6a6b68"). InnerVolumeSpecName "kube-api-access-mqsjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:35:02.811790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:02.811754 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:35:02.811790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:02.811784 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:35:02.811790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:02.811793 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqsjd\" (UniqueName: \"kubernetes.io/projected/e7c73b24-9b5d-4883-9873-8eb1be6a6b68-kube-api-access-mqsjd\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:35:03.406860 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:03.406830 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" Oct 29 23:35:03.407030 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:03.406829 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24" event={"ID":"e7c73b24-9b5d-4883-9873-8eb1be6a6b68","Type":"ContainerDied","Data":"e1cb966cb3dcab126979b0c2fe5f76723800f13e70e01e2f86d6d6c2354d81b1"} Oct 29 23:35:03.407030 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:03.406964 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1cb966cb3dcab126979b0c2fe5f76723800f13e70e01e2f86d6d6c2354d81b1" Oct 29 23:35:10.581924 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:10.581887 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:35:15.331128 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:15.331076 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:35:15.340249 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:35:15.340082 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:35:20.976143 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:20.976105 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv"] Oct 29 23:35:20.976719 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:20.976698 6507 cpu_manager.go:414] "containerMap: removing container" podUID="e7c73b24-9b5d-4883-9873-8eb1be6a6b68" containerName="revoke-keys" Oct 29 23:35:20.976795 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:20.976723 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c73b24-9b5d-4883-9873-8eb1be6a6b68" containerName="revoke-keys" Oct 29 23:35:20.976882 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:20.976869 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c73b24-9b5d-4883-9873-8eb1be6a6b68" containerName="revoke-keys" Oct 29 23:35:20.980012 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:20.979992 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv" Oct 29 23:35:21.001794 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:21.001758 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:35:21.001953 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:21.001878 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:35:21.001953 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:21.001949 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:35:21.002100 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:21.002075 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:35:21.004209 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:21.004183 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv"] Oct 29 23:35:21.033852 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:21.033818 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee900d90-3aab-42a0-a9fd-57602324aae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8n6nv\" (UID: \"ee900d90-3aab-42a0-a9fd-57602324aae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv" Oct 29 23:35:21.034020 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:21.033884 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee900d90-3aab-42a0-a9fd-57602324aae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8n6nv\" (UID: \"ee900d90-3aab-42a0-a9fd-57602324aae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv" Oct 29 23:35:21.034020 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:21.033965 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8bz\" (UniqueName: \"kubernetes.io/projected/ee900d90-3aab-42a0-a9fd-57602324aae6-kube-api-access-lt8bz\") pod \"maas-api-revoke-sub-a4aef7d7-8n6nv\" (UID: \"ee900d90-3aab-42a0-a9fd-57602324aae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv" Oct 29 23:35:21.135408 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:21.135369 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee900d90-3aab-42a0-a9fd-57602324aae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8n6nv\" (UID: \"ee900d90-3aab-42a0-a9fd-57602324aae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv" Oct 29 23:35:21.135615 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:21.135450 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee900d90-3aab-42a0-a9fd-57602324aae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8n6nv\" (UID: \"ee900d90-3aab-42a0-a9fd-57602324aae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv" Oct 29 23:35:21.135615 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:21.135508 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8bz\" (UniqueName: \"kubernetes.io/projected/ee900d90-3aab-42a0-a9fd-57602324aae6-kube-api-access-lt8bz\") pod \"maas-api-revoke-sub-a4aef7d7-8n6nv\" (UID: \"ee900d90-3aab-42a0-a9fd-57602324aae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv" Oct 29 23:35:21.136081 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:21.136057 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee900d90-3aab-42a0-a9fd-57602324aae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8n6nv\" (UID: \"ee900d90-3aab-42a0-a9fd-57602324aae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv" Oct 29 23:35:21.137859 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:21.137832 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee900d90-3aab-42a0-a9fd-57602324aae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-8n6nv\" (UID: \"ee900d90-3aab-42a0-a9fd-57602324aae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv" Oct 29 23:35:21.152879 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:21.152846 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8bz\" (UniqueName: \"kubernetes.io/projected/ee900d90-3aab-42a0-a9fd-57602324aae6-kube-api-access-lt8bz\") pod \"maas-api-revoke-sub-a4aef7d7-8n6nv\" (UID: \"ee900d90-3aab-42a0-a9fd-57602324aae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv" Oct 29 23:35:21.292592 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:21.292509 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv" Oct 29 23:35:21.589378 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:21.589286 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv"] Oct 29 23:35:21.593079 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:35:21.593055 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee900d90_3aab_42a0_a9fd_57602324aae6.slice/crio-cc99141d1fdb0edebcddba14fb9a68e57b9bed5d3c894c472515fcdc4d25ef82 WatchSource:0}: Error finding container cc99141d1fdb0edebcddba14fb9a68e57b9bed5d3c894c472515fcdc4d25ef82: Status 404 returned error can't find the container with id cc99141d1fdb0edebcddba14fb9a68e57b9bed5d3c894c472515fcdc4d25ef82 Oct 29 23:35:22.542567 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:22.542536 6507 generic.go:356] "Generic (PLEG): container finished" podID="ee900d90-3aab-42a0-a9fd-57602324aae6" containerID="7ca5ac9fb1b747a7b219145a015ba340d3ca184d5a56b7a38b3b41069dac32f1" exitCode=0 Oct 29 23:35:22.542915 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:22.542628 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv" event={"ID":"ee900d90-3aab-42a0-a9fd-57602324aae6","Type":"ContainerDied","Data":"7ca5ac9fb1b747a7b219145a015ba340d3ca184d5a56b7a38b3b41069dac32f1"} Oct 29 23:35:22.542915 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:22.542680 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv" event={"ID":"ee900d90-3aab-42a0-a9fd-57602324aae6","Type":"ContainerStarted","Data":"cc99141d1fdb0edebcddba14fb9a68e57b9bed5d3c894c472515fcdc4d25ef82"} Oct 29 23:35:23.679523 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:23.679494 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv" Oct 29 23:35:23.758610 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:23.758574 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee900d90-3aab-42a0-a9fd-57602324aae6-kube-api-access-lt8bz\" (UniqueName: \"kubernetes.io/projected/ee900d90-3aab-42a0-a9fd-57602324aae6-kube-api-access-lt8bz\") pod \"ee900d90-3aab-42a0-a9fd-57602324aae6\" (UID: \"ee900d90-3aab-42a0-a9fd-57602324aae6\") " Oct 29 23:35:23.758814 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:23.758632 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee900d90-3aab-42a0-a9fd-57602324aae6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee900d90-3aab-42a0-a9fd-57602324aae6-maas-api-xks-ca\") pod \"ee900d90-3aab-42a0-a9fd-57602324aae6\" (UID: \"ee900d90-3aab-42a0-a9fd-57602324aae6\") " Oct 29 23:35:23.758814 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:23.758746 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee900d90-3aab-42a0-a9fd-57602324aae6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee900d90-3aab-42a0-a9fd-57602324aae6-maas-api-openshift-ca\") pod \"ee900d90-3aab-42a0-a9fd-57602324aae6\" (UID: \"ee900d90-3aab-42a0-a9fd-57602324aae6\") " Oct 29 23:35:23.759153 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:23.759118 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee900d90-3aab-42a0-a9fd-57602324aae6-maas-api-openshift-ca" pod "ee900d90-3aab-42a0-a9fd-57602324aae6" (UID: "ee900d90-3aab-42a0-a9fd-57602324aae6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:35:23.760674 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:23.760651 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee900d90-3aab-42a0-a9fd-57602324aae6-maas-api-xks-ca" pod "ee900d90-3aab-42a0-a9fd-57602324aae6" (UID: "ee900d90-3aab-42a0-a9fd-57602324aae6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:35:23.760796 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:23.760777 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee900d90-3aab-42a0-a9fd-57602324aae6-kube-api-access-lt8bz" pod "ee900d90-3aab-42a0-a9fd-57602324aae6" (UID: "ee900d90-3aab-42a0-a9fd-57602324aae6"). InnerVolumeSpecName "kube-api-access-lt8bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:35:23.860434 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:23.860348 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8bz\" (UniqueName: \"kubernetes.io/projected/ee900d90-3aab-42a0-a9fd-57602324aae6-kube-api-access-lt8bz\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:35:23.860434 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:23.860380 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee900d90-3aab-42a0-a9fd-57602324aae6-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:35:23.860434 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:23.860390 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee900d90-3aab-42a0-a9fd-57602324aae6-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:35:24.554931 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:24.554896 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv" Oct 29 23:35:24.555114 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:24.554896 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv" event={"ID":"ee900d90-3aab-42a0-a9fd-57602324aae6","Type":"ContainerDied","Data":"cc99141d1fdb0edebcddba14fb9a68e57b9bed5d3c894c472515fcdc4d25ef82"} Oct 29 23:35:24.555114 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:24.555021 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc99141d1fdb0edebcddba14fb9a68e57b9bed5d3c894c472515fcdc4d25ef82" Oct 29 23:35:27.027364 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:27.027328 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp"] Oct 29 23:35:27.035724 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:27.035697 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56c27fd0-gz4sp"] Oct 29 23:35:28.751911 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:28.751879 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aecafa4-7d6e-4fa5-adc8-05e37a772b0e" path="/var/lib/kubelet/pods/4aecafa4-7d6e-4fa5-adc8-05e37a772b0e/volumes" Oct 29 23:35:31.031522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:31.031484 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn"] Oct 29 23:35:31.041552 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:31.041489 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd"] Oct 29 23:35:31.043587 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:31.043554 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63b88d11-kmgxn"] Oct 29 23:35:31.045586 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:31.045558 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aaf95ce-zkkxd"] Oct 29 23:35:32.186276 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:32.186236 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:35:32.753485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:32.752448 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ba8a59-d4e6-4d18-ba32-3c6641e543c4" path="/var/lib/kubelet/pods/60ba8a59-d4e6-4d18-ba32-3c6641e543c4/volumes" Oct 29 23:35:32.753485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:32.753250 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c49368-4334-4699-99dc-a1c19dbc9fce" path="/var/lib/kubelet/pods/73c49368-4334-4699-99dc-a1c19dbc9fce/volumes" Oct 29 23:35:36.690906 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:36.690872 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:35:41.210221 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.210177 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz"] Oct 29 23:35:41.210733 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.210713 6507 cpu_manager.go:414] "containerMap: removing container" podUID="ee900d90-3aab-42a0-a9fd-57602324aae6" containerName="revoke-keys" Oct 29 23:35:41.210781 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.210734 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee900d90-3aab-42a0-a9fd-57602324aae6" containerName="revoke-keys" Oct 29 23:35:41.210836 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.210827 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee900d90-3aab-42a0-a9fd-57602324aae6" containerName="revoke-keys" Oct 29 23:35:41.213889 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.213874 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" Oct 29 23:35:41.235679 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.235625 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:35:41.235866 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.235694 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:35:41.235866 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.235765 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:35:41.236002 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.235643 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:35:41.238014 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.237989 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz"] Oct 29 23:35:41.265162 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.265127 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5f6m\" (UniqueName: \"kubernetes.io/projected/90a46803-d1ee-4689-8030-c2087e98f4ef-kube-api-access-b5f6m\") pod \"maas-api-revoke-sub-79805ca9-rptjz\" (UID: \"90a46803-d1ee-4689-8030-c2087e98f4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" Oct 29 23:35:41.265325 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.265253 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a46803-d1ee-4689-8030-c2087e98f4ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rptjz\" (UID: \"90a46803-d1ee-4689-8030-c2087e98f4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" Oct 29 23:35:41.265325 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.265311 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a46803-d1ee-4689-8030-c2087e98f4ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rptjz\" (UID: \"90a46803-d1ee-4689-8030-c2087e98f4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" Oct 29 23:35:41.366284 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.366238 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a46803-d1ee-4689-8030-c2087e98f4ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rptjz\" (UID: \"90a46803-d1ee-4689-8030-c2087e98f4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" Oct 29 23:35:41.366512 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.366311 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5f6m\" (UniqueName: \"kubernetes.io/projected/90a46803-d1ee-4689-8030-c2087e98f4ef-kube-api-access-b5f6m\") pod \"maas-api-revoke-sub-79805ca9-rptjz\" (UID: \"90a46803-d1ee-4689-8030-c2087e98f4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" Oct 29 23:35:41.366512 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.366360 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a46803-d1ee-4689-8030-c2087e98f4ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rptjz\" (UID: \"90a46803-d1ee-4689-8030-c2087e98f4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" Oct 29 23:35:41.367036 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.367006 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a46803-d1ee-4689-8030-c2087e98f4ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rptjz\" (UID: \"90a46803-d1ee-4689-8030-c2087e98f4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" Oct 29 23:35:41.368652 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.368633 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a46803-d1ee-4689-8030-c2087e98f4ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rptjz\" (UID: \"90a46803-d1ee-4689-8030-c2087e98f4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" Oct 29 23:35:41.384470 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.384439 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5f6m\" (UniqueName: \"kubernetes.io/projected/90a46803-d1ee-4689-8030-c2087e98f4ef-kube-api-access-b5f6m\") pod \"maas-api-revoke-sub-79805ca9-rptjz\" (UID: \"90a46803-d1ee-4689-8030-c2087e98f4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" Oct 29 23:35:41.526877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.526792 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" Oct 29 23:35:41.820660 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:41.820570 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz"] Oct 29 23:35:41.824899 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:35:41.824861 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a46803_d1ee_4689_8030_c2087e98f4ef.slice/crio-598bafa64cef6db549f88ec49e759d224a75ae332038ce48de1cffff8ff75cb6 WatchSource:0}: Error finding container 598bafa64cef6db549f88ec49e759d224a75ae332038ce48de1cffff8ff75cb6: Status 404 returned error can't find the container with id 598bafa64cef6db549f88ec49e759d224a75ae332038ce48de1cffff8ff75cb6 Oct 29 23:35:42.697273 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:42.697232 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" event={"ID":"90a46803-d1ee-4689-8030-c2087e98f4ef","Type":"ContainerStarted","Data":"c283966f8589e4b6b4f10e070650a39a0f1141d16352216d461e5871c9f2ed36"} Oct 29 23:35:42.697777 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:42.697286 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" event={"ID":"90a46803-d1ee-4689-8030-c2087e98f4ef","Type":"ContainerStarted","Data":"598bafa64cef6db549f88ec49e759d224a75ae332038ce48de1cffff8ff75cb6"} Oct 29 23:35:42.731941 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:42.731886 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" podStartSLOduration=1.73187219 podStartE2EDuration="1.73187219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:35:42.730323044 +0000 UTC m=+1312.093327752" watchObservedRunningTime="2026-10-29 23:35:42.73187219 +0000 UTC m=+1312.094876913" Oct 29 23:35:43.710924 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:43.710884 6507 generic.go:356] "Generic (PLEG): container finished" podID="90a46803-d1ee-4689-8030-c2087e98f4ef" containerID="c283966f8589e4b6b4f10e070650a39a0f1141d16352216d461e5871c9f2ed36" exitCode=0 Oct 29 23:35:43.711349 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:43.710945 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" event={"ID":"90a46803-d1ee-4689-8030-c2087e98f4ef","Type":"ContainerDied","Data":"c283966f8589e4b6b4f10e070650a39a0f1141d16352216d461e5871c9f2ed36"} Oct 29 23:35:44.854513 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:44.854484 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" Oct 29 23:35:45.004133 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:45.004043 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a46803-d1ee-4689-8030-c2087e98f4ef-kube-api-access-b5f6m\" (UniqueName: \"kubernetes.io/projected/90a46803-d1ee-4689-8030-c2087e98f4ef-kube-api-access-b5f6m\") pod \"90a46803-d1ee-4689-8030-c2087e98f4ef\" (UID: \"90a46803-d1ee-4689-8030-c2087e98f4ef\") " Oct 29 23:35:45.004133 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:45.004097 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a46803-d1ee-4689-8030-c2087e98f4ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a46803-d1ee-4689-8030-c2087e98f4ef-maas-api-openshift-ca\") pod \"90a46803-d1ee-4689-8030-c2087e98f4ef\" (UID: \"90a46803-d1ee-4689-8030-c2087e98f4ef\") " Oct 29 23:35:45.004377 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:45.004172 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90a46803-d1ee-4689-8030-c2087e98f4ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a46803-d1ee-4689-8030-c2087e98f4ef-maas-api-xks-ca\") pod \"90a46803-d1ee-4689-8030-c2087e98f4ef\" (UID: \"90a46803-d1ee-4689-8030-c2087e98f4ef\") " Oct 29 23:35:45.004506 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:45.004481 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a46803-d1ee-4689-8030-c2087e98f4ef-maas-api-openshift-ca" pod "90a46803-d1ee-4689-8030-c2087e98f4ef" (UID: "90a46803-d1ee-4689-8030-c2087e98f4ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:35:45.006339 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:45.006313 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90a46803-d1ee-4689-8030-c2087e98f4ef-maas-api-xks-ca" pod "90a46803-d1ee-4689-8030-c2087e98f4ef" (UID: "90a46803-d1ee-4689-8030-c2087e98f4ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:35:45.006437 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:45.006319 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a46803-d1ee-4689-8030-c2087e98f4ef-kube-api-access-b5f6m" pod "90a46803-d1ee-4689-8030-c2087e98f4ef" (UID: "90a46803-d1ee-4689-8030-c2087e98f4ef"). InnerVolumeSpecName "kube-api-access-b5f6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:35:45.105436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:45.105398 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5f6m\" (UniqueName: \"kubernetes.io/projected/90a46803-d1ee-4689-8030-c2087e98f4ef-kube-api-access-b5f6m\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:35:45.105436 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:45.105433 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a46803-d1ee-4689-8030-c2087e98f4ef-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:35:45.105671 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:45.105447 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a46803-d1ee-4689-8030-c2087e98f4ef-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:35:45.224173 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:35:45.224128 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:35:45.725731 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:45.725702 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" Oct 29 23:35:45.725731 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:45.725709 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz" event={"ID":"90a46803-d1ee-4689-8030-c2087e98f4ef","Type":"ContainerDied","Data":"598bafa64cef6db549f88ec49e759d224a75ae332038ce48de1cffff8ff75cb6"} Oct 29 23:35:45.725982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:45.725753 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="598bafa64cef6db549f88ec49e759d224a75ae332038ce48de1cffff8ff75cb6" Oct 29 23:35:52.374285 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:52.374237 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:35:52.893759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:52.893720 6507 scope.go:122] "RemoveContainer" containerID="93a5ded4e82bc6e030b35a0c6d398956d4db5b5eeba214c48a82f482c8ad7171" Oct 29 23:35:52.903511 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:52.903488 6507 scope.go:122] "RemoveContainer" containerID="fef7f17b0b6fa8017c379daa3268b488f0a8754d31eccd67d702b720a04290d9" Oct 29 23:35:52.913496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:52.913447 6507 scope.go:122] "RemoveContainer" containerID="00ba596edceac4f7cd5e8df8d746a8e4b8adef4ca0117c732f3c39b924019f0a" Oct 29 23:35:52.923099 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:52.923074 6507 scope.go:122] "RemoveContainer" containerID="2da4af94c242845006097def6d19755b859e6c94ec51fdec7bcbc3c10fcb0135" Oct 29 23:35:52.932131 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:52.932106 6507 scope.go:122] "RemoveContainer" containerID="c5444b51cda3df973a081208848f7b551c7d5dcf31eea78439d42585c0edd661" Oct 29 23:35:52.943271 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:52.943250 6507 scope.go:122] "RemoveContainer" containerID="5295b915ce26580b1e008566ede142f9dd99d763edc096f2c3062c3c4137d298" Oct 29 23:35:52.951677 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:52.951653 6507 scope.go:122] "RemoveContainer" containerID="161fc1f1be53862ceb08a106094348c1d0f2bac83b48164cbbe6f3631450604c" Oct 29 23:35:56.715482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:35:56.715072 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:36:01.309096 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.309054 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw"] Oct 29 23:36:01.310024 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.309995 6507 cpu_manager.go:414] "containerMap: removing container" podUID="90a46803-d1ee-4689-8030-c2087e98f4ef" containerName="revoke-keys" Oct 29 23:36:01.310024 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.310021 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a46803-d1ee-4689-8030-c2087e98f4ef" containerName="revoke-keys" Oct 29 23:36:01.310182 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.310112 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a46803-d1ee-4689-8030-c2087e98f4ef" containerName="revoke-keys" Oct 29 23:36:01.315224 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.315201 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw" Oct 29 23:36:01.339551 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.339521 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:36:01.339750 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.339690 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:36:01.339750 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.339717 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:36:01.339750 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.339722 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:36:01.342911 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.342882 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw"] Oct 29 23:36:01.399765 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.399730 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d88e96d-992e-4d89-9501-b9cbf8722f66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-675nw\" (UID: \"0d88e96d-992e-4d89-9501-b9cbf8722f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw" Oct 29 23:36:01.399932 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.399781 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d88e96d-992e-4d89-9501-b9cbf8722f66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-675nw\" (UID: \"0d88e96d-992e-4d89-9501-b9cbf8722f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw" Oct 29 23:36:01.399976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.399925 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5s7v\" (UniqueName: \"kubernetes.io/projected/0d88e96d-992e-4d89-9501-b9cbf8722f66-kube-api-access-h5s7v\") pod \"maas-api-revoke-sub-0b7e8edb-675nw\" (UID: \"0d88e96d-992e-4d89-9501-b9cbf8722f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw" Oct 29 23:36:01.501135 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.501094 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5s7v\" (UniqueName: \"kubernetes.io/projected/0d88e96d-992e-4d89-9501-b9cbf8722f66-kube-api-access-h5s7v\") pod \"maas-api-revoke-sub-0b7e8edb-675nw\" (UID: \"0d88e96d-992e-4d89-9501-b9cbf8722f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw" Oct 29 23:36:01.501316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.501149 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d88e96d-992e-4d89-9501-b9cbf8722f66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-675nw\" (UID: \"0d88e96d-992e-4d89-9501-b9cbf8722f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw" Oct 29 23:36:01.501316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.501188 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d88e96d-992e-4d89-9501-b9cbf8722f66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-675nw\" (UID: \"0d88e96d-992e-4d89-9501-b9cbf8722f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw" Oct 29 23:36:01.501843 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.501823 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d88e96d-992e-4d89-9501-b9cbf8722f66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-675nw\" (UID: \"0d88e96d-992e-4d89-9501-b9cbf8722f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw" Oct 29 23:36:01.503691 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.503658 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d88e96d-992e-4d89-9501-b9cbf8722f66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-675nw\" (UID: \"0d88e96d-992e-4d89-9501-b9cbf8722f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw" Oct 29 23:36:01.518947 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.518918 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5s7v\" (UniqueName: \"kubernetes.io/projected/0d88e96d-992e-4d89-9501-b9cbf8722f66-kube-api-access-h5s7v\") pod \"maas-api-revoke-sub-0b7e8edb-675nw\" (UID: \"0d88e96d-992e-4d89-9501-b9cbf8722f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw" Oct 29 23:36:01.628701 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.628614 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw" Oct 29 23:36:01.924497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:01.924379 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw"] Oct 29 23:36:01.928636 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:36:01.928594 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d88e96d_992e_4d89_9501_b9cbf8722f66.slice/crio-4f2abb447e3bb73d44036d0bc25a95a7410e9e41f30c61bfc53fcccf1a11e189 WatchSource:0}: Error finding container 4f2abb447e3bb73d44036d0bc25a95a7410e9e41f30c61bfc53fcccf1a11e189: Status 404 returned error can't find the container with id 4f2abb447e3bb73d44036d0bc25a95a7410e9e41f30c61bfc53fcccf1a11e189 Oct 29 23:36:02.849025 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:02.848990 6507 generic.go:356] "Generic (PLEG): container finished" podID="0d88e96d-992e-4d89-9501-b9cbf8722f66" containerID="596f2763005111626621801eb8a273516a0255858e659afbea6074afee47718f" exitCode=0 Oct 29 23:36:02.849449 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:02.849076 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw" event={"ID":"0d88e96d-992e-4d89-9501-b9cbf8722f66","Type":"ContainerDied","Data":"596f2763005111626621801eb8a273516a0255858e659afbea6074afee47718f"} Oct 29 23:36:02.849449 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:02.849128 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw" event={"ID":"0d88e96d-992e-4d89-9501-b9cbf8722f66","Type":"ContainerStarted","Data":"4f2abb447e3bb73d44036d0bc25a95a7410e9e41f30c61bfc53fcccf1a11e189"} Oct 29 23:36:03.991681 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:03.991656 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw" Oct 29 23:36:04.023897 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:04.023867 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d88e96d-992e-4d89-9501-b9cbf8722f66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d88e96d-992e-4d89-9501-b9cbf8722f66-maas-api-openshift-ca\") pod \"0d88e96d-992e-4d89-9501-b9cbf8722f66\" (UID: \"0d88e96d-992e-4d89-9501-b9cbf8722f66\") " Oct 29 23:36:04.024078 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:04.023974 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d88e96d-992e-4d89-9501-b9cbf8722f66-kube-api-access-h5s7v\" (UniqueName: \"kubernetes.io/projected/0d88e96d-992e-4d89-9501-b9cbf8722f66-kube-api-access-h5s7v\") pod \"0d88e96d-992e-4d89-9501-b9cbf8722f66\" (UID: \"0d88e96d-992e-4d89-9501-b9cbf8722f66\") " Oct 29 23:36:04.024078 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:04.023999 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d88e96d-992e-4d89-9501-b9cbf8722f66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d88e96d-992e-4d89-9501-b9cbf8722f66-maas-api-xks-ca\") pod \"0d88e96d-992e-4d89-9501-b9cbf8722f66\" (UID: \"0d88e96d-992e-4d89-9501-b9cbf8722f66\") " Oct 29 23:36:04.024233 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:04.024207 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d88e96d-992e-4d89-9501-b9cbf8722f66-maas-api-openshift-ca" pod "0d88e96d-992e-4d89-9501-b9cbf8722f66" (UID: "0d88e96d-992e-4d89-9501-b9cbf8722f66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:36:04.026257 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:04.026229 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d88e96d-992e-4d89-9501-b9cbf8722f66-kube-api-access-h5s7v" pod "0d88e96d-992e-4d89-9501-b9cbf8722f66" (UID: "0d88e96d-992e-4d89-9501-b9cbf8722f66"). InnerVolumeSpecName "kube-api-access-h5s7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:36:04.026927 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:04.026303 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d88e96d-992e-4d89-9501-b9cbf8722f66-maas-api-xks-ca" pod "0d88e96d-992e-4d89-9501-b9cbf8722f66" (UID: "0d88e96d-992e-4d89-9501-b9cbf8722f66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:36:04.125027 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:04.124937 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5s7v\" (UniqueName: \"kubernetes.io/projected/0d88e96d-992e-4d89-9501-b9cbf8722f66-kube-api-access-h5s7v\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:36:04.125027 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:04.124971 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d88e96d-992e-4d89-9501-b9cbf8722f66-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:36:04.125027 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:04.124983 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d88e96d-992e-4d89-9501-b9cbf8722f66-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:36:04.861299 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:04.861271 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw" Oct 29 23:36:04.861478 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:04.861308 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw" event={"ID":"0d88e96d-992e-4d89-9501-b9cbf8722f66","Type":"ContainerDied","Data":"4f2abb447e3bb73d44036d0bc25a95a7410e9e41f30c61bfc53fcccf1a11e189"} Oct 29 23:36:04.861478 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:04.861351 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f2abb447e3bb73d44036d0bc25a95a7410e9e41f30c61bfc53fcccf1a11e189" Oct 29 23:36:12.103497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.097247 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28"] Oct 29 23:36:12.103497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.097794 6507 cpu_manager.go:414] "containerMap: removing container" podUID="0d88e96d-992e-4d89-9501-b9cbf8722f66" containerName="revoke-keys" Oct 29 23:36:12.103497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.097815 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d88e96d-992e-4d89-9501-b9cbf8722f66" containerName="revoke-keys" Oct 29 23:36:12.103497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.100350 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d88e96d-992e-4d89-9501-b9cbf8722f66" containerName="revoke-keys" Oct 29 23:36:12.105588 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.105552 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28" Oct 29 23:36:12.128037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.127998 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:36:12.128755 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.128731 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:36:12.129566 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.129381 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:36:12.129669 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.129437 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:36:12.132024 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.132000 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28"] Oct 29 23:36:12.201713 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.201671 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b1d1af3-964b-4411-977e-0477d1b46d86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m2d28\" (UID: \"7b1d1af3-964b-4411-977e-0477d1b46d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28" Oct 29 23:36:12.201910 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.201883 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5d48\" (UniqueName: \"kubernetes.io/projected/7b1d1af3-964b-4411-977e-0477d1b46d86-kube-api-access-k5d48\") pod \"maas-api-revoke-sub-3d4156f1-m2d28\" (UID: \"7b1d1af3-964b-4411-977e-0477d1b46d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28" Oct 29 23:36:12.201955 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.201936 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b1d1af3-964b-4411-977e-0477d1b46d86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m2d28\" (UID: \"7b1d1af3-964b-4411-977e-0477d1b46d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28" Oct 29 23:36:12.303195 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.303149 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5d48\" (UniqueName: \"kubernetes.io/projected/7b1d1af3-964b-4411-977e-0477d1b46d86-kube-api-access-k5d48\") pod \"maas-api-revoke-sub-3d4156f1-m2d28\" (UID: \"7b1d1af3-964b-4411-977e-0477d1b46d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28" Oct 29 23:36:12.303195 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.303200 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b1d1af3-964b-4411-977e-0477d1b46d86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m2d28\" (UID: \"7b1d1af3-964b-4411-977e-0477d1b46d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28" Oct 29 23:36:12.303500 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.303238 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b1d1af3-964b-4411-977e-0477d1b46d86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m2d28\" (UID: \"7b1d1af3-964b-4411-977e-0477d1b46d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28" Oct 29 23:36:12.303983 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.303958 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b1d1af3-964b-4411-977e-0477d1b46d86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m2d28\" (UID: \"7b1d1af3-964b-4411-977e-0477d1b46d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28" Oct 29 23:36:12.305558 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.305530 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b1d1af3-964b-4411-977e-0477d1b46d86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m2d28\" (UID: \"7b1d1af3-964b-4411-977e-0477d1b46d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28" Oct 29 23:36:12.320120 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.320095 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5d48\" (UniqueName: \"kubernetes.io/projected/7b1d1af3-964b-4411-977e-0477d1b46d86-kube-api-access-k5d48\") pod \"maas-api-revoke-sub-3d4156f1-m2d28\" (UID: \"7b1d1af3-964b-4411-977e-0477d1b46d86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28" Oct 29 23:36:12.420356 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.420280 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28" Oct 29 23:36:12.432329 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.432300 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:36:12.715038 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.715006 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28"] Oct 29 23:36:12.719382 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:36:12.719354 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1d1af3_964b_4411_977e_0477d1b46d86.slice/crio-c90720bf12f32b091b694fd2a828172b53f448592c02206f04bbf9aadabea7b4 WatchSource:0}: Error finding container c90720bf12f32b091b694fd2a828172b53f448592c02206f04bbf9aadabea7b4: Status 404 returned error can't find the container with id c90720bf12f32b091b694fd2a828172b53f448592c02206f04bbf9aadabea7b4 Oct 29 23:36:12.912310 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:12.912261 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28" event={"ID":"7b1d1af3-964b-4411-977e-0477d1b46d86","Type":"ContainerStarted","Data":"c90720bf12f32b091b694fd2a828172b53f448592c02206f04bbf9aadabea7b4"} Oct 29 23:36:13.919412 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:13.919321 6507 generic.go:356] "Generic (PLEG): container finished" podID="7b1d1af3-964b-4411-977e-0477d1b46d86" containerID="9e75cc8d32d4153178a7a10018d03b5ebb7834669119ad37cb4fa1d8f988362c" exitCode=0 Oct 29 23:36:13.919825 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:13.919409 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28" event={"ID":"7b1d1af3-964b-4411-977e-0477d1b46d86","Type":"ContainerDied","Data":"9e75cc8d32d4153178a7a10018d03b5ebb7834669119ad37cb4fa1d8f988362c"} Oct 29 23:36:14.024937 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:14.024902 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8"] Oct 29 23:36:14.029293 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:14.029267 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d886a5-v2cw8"] Oct 29 23:36:14.751388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:14.751348 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9" path="/var/lib/kubelet/pods/8623ac86-e80b-4c9c-bc3d-73c5ef5b70f9/volumes" Oct 29 23:36:15.062675 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:15.062651 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28" Oct 29 23:36:15.131871 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:15.131837 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b1d1af3-964b-4411-977e-0477d1b46d86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b1d1af3-964b-4411-977e-0477d1b46d86-maas-api-openshift-ca\") pod \"7b1d1af3-964b-4411-977e-0477d1b46d86\" (UID: \"7b1d1af3-964b-4411-977e-0477d1b46d86\") " Oct 29 23:36:15.132042 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:15.131900 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b1d1af3-964b-4411-977e-0477d1b46d86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b1d1af3-964b-4411-977e-0477d1b46d86-maas-api-xks-ca\") pod \"7b1d1af3-964b-4411-977e-0477d1b46d86\" (UID: \"7b1d1af3-964b-4411-977e-0477d1b46d86\") " Oct 29 23:36:15.132042 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:15.132023 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b1d1af3-964b-4411-977e-0477d1b46d86-kube-api-access-k5d48\" (UniqueName: \"kubernetes.io/projected/7b1d1af3-964b-4411-977e-0477d1b46d86-kube-api-access-k5d48\") pod \"7b1d1af3-964b-4411-977e-0477d1b46d86\" (UID: \"7b1d1af3-964b-4411-977e-0477d1b46d86\") " Oct 29 23:36:15.132294 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:15.132262 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1d1af3-964b-4411-977e-0477d1b46d86-maas-api-openshift-ca" pod "7b1d1af3-964b-4411-977e-0477d1b46d86" (UID: "7b1d1af3-964b-4411-977e-0477d1b46d86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:36:15.132509 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:15.132486 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b1d1af3-964b-4411-977e-0477d1b46d86-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:36:15.134192 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:15.134165 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b1d1af3-964b-4411-977e-0477d1b46d86-maas-api-xks-ca" pod "7b1d1af3-964b-4411-977e-0477d1b46d86" (UID: "7b1d1af3-964b-4411-977e-0477d1b46d86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:36:15.134310 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:15.134208 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1d1af3-964b-4411-977e-0477d1b46d86-kube-api-access-k5d48" pod "7b1d1af3-964b-4411-977e-0477d1b46d86" (UID: "7b1d1af3-964b-4411-977e-0477d1b46d86"). InnerVolumeSpecName "kube-api-access-k5d48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:36:15.222261 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:36:15.222223 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:36:15.234009 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:15.233985 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b1d1af3-964b-4411-977e-0477d1b46d86-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:36:15.234009 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:15.234007 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5d48\" (UniqueName: \"kubernetes.io/projected/7b1d1af3-964b-4411-977e-0477d1b46d86-kube-api-access-k5d48\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:36:15.931956 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:15.931923 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28" Oct 29 23:36:15.932138 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:15.931949 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28" event={"ID":"7b1d1af3-964b-4411-977e-0477d1b46d86","Type":"ContainerDied","Data":"c90720bf12f32b091b694fd2a828172b53f448592c02206f04bbf9aadabea7b4"} Oct 29 23:36:15.932138 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:15.931999 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c90720bf12f32b091b694fd2a828172b53f448592c02206f04bbf9aadabea7b4" Oct 29 23:36:17.032626 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:17.032589 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r"] Oct 29 23:36:17.042259 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:17.042232 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4832c42-bv59r"] Oct 29 23:36:18.751956 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:18.751922 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="175a760e-0d42-42e9-a626-890e1cb9bc27" path="/var/lib/kubelet/pods/175a760e-0d42-42e9-a626-890e1cb9bc27/volumes" Oct 29 23:36:23.268286 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:23.268251 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:36:28.386715 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:28.386672 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:36:33.444501 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.444366 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28"] Oct 29 23:36:33.453270 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.453242 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m2d28"] Oct 29 23:36:33.496932 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.496882 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl"] Oct 29 23:36:33.497469 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.497438 6507 cpu_manager.go:414] "containerMap: removing container" podUID="7b1d1af3-964b-4411-977e-0477d1b46d86" containerName="revoke-keys" Oct 29 23:36:33.497547 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.497469 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b1d1af3-964b-4411-977e-0477d1b46d86" containerName="revoke-keys" Oct 29 23:36:33.497591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.497552 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b1d1af3-964b-4411-977e-0477d1b46d86" containerName="revoke-keys" Oct 29 23:36:33.501971 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.501952 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl" Oct 29 23:36:33.514131 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.514108 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:36:33.514131 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.514118 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:36:33.514362 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.514342 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:36:33.514435 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.514361 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:36:33.521180 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.521077 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl"] Oct 29 23:36:33.653674 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.653636 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2m4gl\" (UID: \"4f8205fe-972d-4dc3-a08c-ddce4657ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl" Oct 29 23:36:33.653674 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.653671 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kknml\" (UniqueName: \"kubernetes.io/projected/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-kube-api-access-kknml\") pod \"maas-api-revoke-sub-3d4156f1-2m4gl\" (UID: \"4f8205fe-972d-4dc3-a08c-ddce4657ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl" Oct 29 23:36:33.653936 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.653699 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2m4gl\" (UID: \"4f8205fe-972d-4dc3-a08c-ddce4657ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl" Oct 29 23:36:33.755055 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.754963 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2m4gl\" (UID: \"4f8205fe-972d-4dc3-a08c-ddce4657ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl" Oct 29 23:36:33.755055 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.754999 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kknml\" (UniqueName: \"kubernetes.io/projected/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-kube-api-access-kknml\") pod \"maas-api-revoke-sub-3d4156f1-2m4gl\" (UID: \"4f8205fe-972d-4dc3-a08c-ddce4657ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl" Oct 29 23:36:33.755055 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.755023 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2m4gl\" (UID: \"4f8205fe-972d-4dc3-a08c-ddce4657ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl" Oct 29 23:36:33.755708 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.755684 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2m4gl\" (UID: \"4f8205fe-972d-4dc3-a08c-ddce4657ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl" Oct 29 23:36:33.757336 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.757315 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2m4gl\" (UID: \"4f8205fe-972d-4dc3-a08c-ddce4657ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl" Oct 29 23:36:33.771838 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.771808 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kknml\" (UniqueName: \"kubernetes.io/projected/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-kube-api-access-kknml\") pod \"maas-api-revoke-sub-3d4156f1-2m4gl\" (UID: \"4f8205fe-972d-4dc3-a08c-ddce4657ab1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl" Oct 29 23:36:33.814694 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:33.814656 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl" Oct 29 23:36:34.110881 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:34.110789 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl"] Oct 29 23:36:34.115226 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:36:34.115196 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8205fe_972d_4dc3_a08c_ddce4657ab1a.slice/crio-df9062209a9d6d94bdd620a3ea251f17cfa26fd81c2d66e3e100a9429336367a WatchSource:0}: Error finding container df9062209a9d6d94bdd620a3ea251f17cfa26fd81c2d66e3e100a9429336367a: Status 404 returned error can't find the container with id df9062209a9d6d94bdd620a3ea251f17cfa26fd81c2d66e3e100a9429336367a Oct 29 23:36:34.752038 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:34.752007 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b1d1af3-964b-4411-977e-0477d1b46d86" path="/var/lib/kubelet/pods/7b1d1af3-964b-4411-977e-0477d1b46d86/volumes" Oct 29 23:36:35.075813 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:35.075780 6507 generic.go:356] "Generic (PLEG): container finished" podID="4f8205fe-972d-4dc3-a08c-ddce4657ab1a" containerID="30a5a71a6e24210c5b8df2eaf5ee9a531008f9922fe186ec9709378b1c4eeb5a" exitCode=0 Oct 29 23:36:35.075976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:35.075866 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl" event={"ID":"4f8205fe-972d-4dc3-a08c-ddce4657ab1a","Type":"ContainerDied","Data":"30a5a71a6e24210c5b8df2eaf5ee9a531008f9922fe186ec9709378b1c4eeb5a"} Oct 29 23:36:35.075976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:35.075926 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl" event={"ID":"4f8205fe-972d-4dc3-a08c-ddce4657ab1a","Type":"ContainerStarted","Data":"df9062209a9d6d94bdd620a3ea251f17cfa26fd81c2d66e3e100a9429336367a"} Oct 29 23:36:36.216372 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:36.216346 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl" Oct 29 23:36:36.279737 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:36.279708 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-kube-api-access-kknml\" (UniqueName: \"kubernetes.io/projected/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-kube-api-access-kknml\") pod \"4f8205fe-972d-4dc3-a08c-ddce4657ab1a\" (UID: \"4f8205fe-972d-4dc3-a08c-ddce4657ab1a\") " Oct 29 23:36:36.279919 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:36.279756 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-maas-api-xks-ca\") pod \"4f8205fe-972d-4dc3-a08c-ddce4657ab1a\" (UID: \"4f8205fe-972d-4dc3-a08c-ddce4657ab1a\") " Oct 29 23:36:36.279919 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:36.279806 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-maas-api-openshift-ca\") pod \"4f8205fe-972d-4dc3-a08c-ddce4657ab1a\" (UID: \"4f8205fe-972d-4dc3-a08c-ddce4657ab1a\") " Oct 29 23:36:36.280277 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:36.280248 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-maas-api-openshift-ca" pod "4f8205fe-972d-4dc3-a08c-ddce4657ab1a" (UID: "4f8205fe-972d-4dc3-a08c-ddce4657ab1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:36:36.281838 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:36.281809 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-maas-api-xks-ca" pod "4f8205fe-972d-4dc3-a08c-ddce4657ab1a" (UID: "4f8205fe-972d-4dc3-a08c-ddce4657ab1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:36:36.281986 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:36.281965 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-kube-api-access-kknml" pod "4f8205fe-972d-4dc3-a08c-ddce4657ab1a" (UID: "4f8205fe-972d-4dc3-a08c-ddce4657ab1a"). InnerVolumeSpecName "kube-api-access-kknml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:36:36.381389 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:36.381302 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:36:36.381389 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:36.381336 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:36:36.381389 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:36.381345 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kknml\" (UniqueName: \"kubernetes.io/projected/4f8205fe-972d-4dc3-a08c-ddce4657ab1a-kube-api-access-kknml\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:36:37.055542 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:37.055440 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h"] Oct 29 23:36:37.065115 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:37.065085 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc0f98f9-hxv9h"] Oct 29 23:36:37.088583 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:37.088553 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl" Oct 29 23:36:37.088763 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:37.088581 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl" event={"ID":"4f8205fe-972d-4dc3-a08c-ddce4657ab1a","Type":"ContainerDied","Data":"df9062209a9d6d94bdd620a3ea251f17cfa26fd81c2d66e3e100a9429336367a"} Oct 29 23:36:37.088763 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:37.088666 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df9062209a9d6d94bdd620a3ea251f17cfa26fd81c2d66e3e100a9429336367a" Oct 29 23:36:38.751538 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:38.751496 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d07b2cd-119b-4fdb-a067-7adeec5d5b2c" path="/var/lib/kubelet/pods/9d07b2cd-119b-4fdb-a067-7adeec5d5b2c/volumes" Oct 29 23:36:45.222894 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:36:45.222856 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:36:50.901196 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:50.901155 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl"] Oct 29 23:36:50.909993 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:50.909968 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2m4gl"] Oct 29 23:36:50.948037 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:50.947994 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx"] Oct 29 23:36:50.948646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:50.948625 6507 cpu_manager.go:414] "containerMap: removing container" podUID="4f8205fe-972d-4dc3-a08c-ddce4657ab1a" containerName="revoke-keys" Oct 29 23:36:50.948646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:50.948646 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f8205fe-972d-4dc3-a08c-ddce4657ab1a" containerName="revoke-keys" Oct 29 23:36:50.948797 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:50.948705 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f8205fe-972d-4dc3-a08c-ddce4657ab1a" containerName="revoke-keys" Oct 29 23:36:50.952020 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:50.952004 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" Oct 29 23:36:50.963891 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:50.963868 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:36:50.963998 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:50.963982 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:36:50.964052 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:50.963989 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:36:50.964129 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:50.964109 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:36:50.972154 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:50.972129 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx"] Oct 29 23:36:51.023851 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:51.023816 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a971a987-be36-49fe-bf46-1740b74988c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m7fzx\" (UID: \"a971a987-be36-49fe-bf46-1740b74988c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" Oct 29 23:36:51.024030 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:51.023932 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnshp\" (UniqueName: \"kubernetes.io/projected/a971a987-be36-49fe-bf46-1740b74988c1-kube-api-access-nnshp\") pod \"maas-api-revoke-sub-3d4156f1-m7fzx\" (UID: \"a971a987-be36-49fe-bf46-1740b74988c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" Oct 29 23:36:51.024030 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:51.023981 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a971a987-be36-49fe-bf46-1740b74988c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m7fzx\" (UID: \"a971a987-be36-49fe-bf46-1740b74988c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" Oct 29 23:36:51.125242 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:51.125205 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a971a987-be36-49fe-bf46-1740b74988c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m7fzx\" (UID: \"a971a987-be36-49fe-bf46-1740b74988c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" Oct 29 23:36:51.125451 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:51.125263 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a971a987-be36-49fe-bf46-1740b74988c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m7fzx\" (UID: \"a971a987-be36-49fe-bf46-1740b74988c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" Oct 29 23:36:51.125451 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:51.125433 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnshp\" (UniqueName: \"kubernetes.io/projected/a971a987-be36-49fe-bf46-1740b74988c1-kube-api-access-nnshp\") pod \"maas-api-revoke-sub-3d4156f1-m7fzx\" (UID: \"a971a987-be36-49fe-bf46-1740b74988c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" Oct 29 23:36:51.125957 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:51.125935 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a971a987-be36-49fe-bf46-1740b74988c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m7fzx\" (UID: \"a971a987-be36-49fe-bf46-1740b74988c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" Oct 29 23:36:51.127728 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:51.127705 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a971a987-be36-49fe-bf46-1740b74988c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m7fzx\" (UID: \"a971a987-be36-49fe-bf46-1740b74988c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" Oct 29 23:36:51.142291 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:51.142271 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnshp\" (UniqueName: \"kubernetes.io/projected/a971a987-be36-49fe-bf46-1740b74988c1-kube-api-access-nnshp\") pod \"maas-api-revoke-sub-3d4156f1-m7fzx\" (UID: \"a971a987-be36-49fe-bf46-1740b74988c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" Oct 29 23:36:51.264659 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:51.264622 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" Oct 29 23:36:51.559526 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:51.559428 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx"] Oct 29 23:36:51.563429 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:36:51.563401 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda971a987_be36_49fe_bf46_1740b74988c1.slice/crio-254d3a457fb73d16991c4a99fa50e49be8d60d40f98bc1897b171098dcf571ed WatchSource:0}: Error finding container 254d3a457fb73d16991c4a99fa50e49be8d60d40f98bc1897b171098dcf571ed: Status 404 returned error can't find the container with id 254d3a457fb73d16991c4a99fa50e49be8d60d40f98bc1897b171098dcf571ed Oct 29 23:36:52.026573 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:52.026539 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn"] Oct 29 23:36:52.035482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:52.035441 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0df1f2-qlvvn"] Oct 29 23:36:52.173178 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:52.173085 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" event={"ID":"a971a987-be36-49fe-bf46-1740b74988c1","Type":"ContainerStarted","Data":"abbaf0f855fac1e04cc8b3217b6a4515ab5d3308ea6939e7e5b3cf09a6fd68cd"} Oct 29 23:36:52.173322 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:52.173182 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" event={"ID":"a971a987-be36-49fe-bf46-1740b74988c1","Type":"ContainerStarted","Data":"254d3a457fb73d16991c4a99fa50e49be8d60d40f98bc1897b171098dcf571ed"} Oct 29 23:36:52.206718 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:52.206649 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" podStartSLOduration=2.206633907 podStartE2EDuration="2.206633907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:36:52.205549027 +0000 UTC m=+1381.568553749" watchObservedRunningTime="2026-10-29 23:36:52.206633907 +0000 UTC m=+1381.569638629" Oct 29 23:36:52.752139 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:52.752105 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a425f6-b9c8-40a4-a403-fbd3a6d7baa4" path="/var/lib/kubelet/pods/22a425f6-b9c8-40a4-a403-fbd3a6d7baa4/volumes" Oct 29 23:36:52.752606 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:52.752585 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f8205fe-972d-4dc3-a08c-ddce4657ab1a" path="/var/lib/kubelet/pods/4f8205fe-972d-4dc3-a08c-ddce4657ab1a/volumes" Oct 29 23:36:52.997625 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:52.997596 6507 scope.go:122] "RemoveContainer" containerID="14405dd609be90b271fed73afb5b079e590a5ea82518ed060c3ca30390e01548" Oct 29 23:36:53.006574 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:53.006548 6507 scope.go:122] "RemoveContainer" containerID="77168e06a14518aae60f8718992b888a0791e1b37eda1cdb613dbd77fc9588ce" Oct 29 23:36:53.015003 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:53.014981 6507 scope.go:122] "RemoveContainer" containerID="367c8108bda185276353053b9b26e69d7aa717f04968b82982b4c4ac99c6d149" Oct 29 23:36:53.022721 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:53.022697 6507 scope.go:122] "RemoveContainer" containerID="b9b9a0ff88ef18df461ff7699668334ca12a188bfb1619ca12e5e736f3c684d0" Oct 29 23:36:53.032235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:53.032214 6507 scope.go:122] "RemoveContainer" containerID="e20754ef5d4677ddac128d905bb199243b78f2bdb324b1a08ea1fc9352c1f0cf" Oct 29 23:36:53.179772 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:53.179736 6507 generic.go:356] "Generic (PLEG): container finished" podID="a971a987-be36-49fe-bf46-1740b74988c1" containerID="abbaf0f855fac1e04cc8b3217b6a4515ab5d3308ea6939e7e5b3cf09a6fd68cd" exitCode=0 Oct 29 23:36:53.179954 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:53.179779 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" event={"ID":"a971a987-be36-49fe-bf46-1740b74988c1","Type":"ContainerDied","Data":"abbaf0f855fac1e04cc8b3217b6a4515ab5d3308ea6939e7e5b3cf09a6fd68cd"} Oct 29 23:36:54.316057 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:54.316033 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" Oct 29 23:36:54.455449 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:54.455360 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a971a987-be36-49fe-bf46-1740b74988c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a971a987-be36-49fe-bf46-1740b74988c1-maas-api-openshift-ca\") pod \"a971a987-be36-49fe-bf46-1740b74988c1\" (UID: \"a971a987-be36-49fe-bf46-1740b74988c1\") " Oct 29 23:36:54.455449 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:54.455398 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a971a987-be36-49fe-bf46-1740b74988c1-kube-api-access-nnshp\" (UniqueName: \"kubernetes.io/projected/a971a987-be36-49fe-bf46-1740b74988c1-kube-api-access-nnshp\") pod \"a971a987-be36-49fe-bf46-1740b74988c1\" (UID: \"a971a987-be36-49fe-bf46-1740b74988c1\") " Oct 29 23:36:54.455449 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:54.455440 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a971a987-be36-49fe-bf46-1740b74988c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a971a987-be36-49fe-bf46-1740b74988c1-maas-api-xks-ca\") pod \"a971a987-be36-49fe-bf46-1740b74988c1\" (UID: \"a971a987-be36-49fe-bf46-1740b74988c1\") " Oct 29 23:36:54.455831 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:54.455807 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a971a987-be36-49fe-bf46-1740b74988c1-maas-api-openshift-ca" pod "a971a987-be36-49fe-bf46-1740b74988c1" (UID: "a971a987-be36-49fe-bf46-1740b74988c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:36:54.457584 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:54.457552 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a971a987-be36-49fe-bf46-1740b74988c1-maas-api-xks-ca" pod "a971a987-be36-49fe-bf46-1740b74988c1" (UID: "a971a987-be36-49fe-bf46-1740b74988c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:36:54.457693 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:54.457641 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a971a987-be36-49fe-bf46-1740b74988c1-kube-api-access-nnshp" pod "a971a987-be36-49fe-bf46-1740b74988c1" (UID: "a971a987-be36-49fe-bf46-1740b74988c1"). InnerVolumeSpecName "kube-api-access-nnshp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:36:54.557068 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:54.557032 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a971a987-be36-49fe-bf46-1740b74988c1-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:36:54.557068 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:54.557063 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnshp\" (UniqueName: \"kubernetes.io/projected/a971a987-be36-49fe-bf46-1740b74988c1-kube-api-access-nnshp\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:36:54.557068 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:54.557074 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a971a987-be36-49fe-bf46-1740b74988c1-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:36:55.192357 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:55.192330 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" Oct 29 23:36:55.192357 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:55.192345 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx" event={"ID":"a971a987-be36-49fe-bf46-1740b74988c1","Type":"ContainerDied","Data":"254d3a457fb73d16991c4a99fa50e49be8d60d40f98bc1897b171098dcf571ed"} Oct 29 23:36:55.192598 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:36:55.192399 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="254d3a457fb73d16991c4a99fa50e49be8d60d40f98bc1897b171098dcf571ed" Oct 29 23:37:06.672033 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.671987 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl"] Oct 29 23:37:06.672794 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.672574 6507 cpu_manager.go:414] "containerMap: removing container" podUID="a971a987-be36-49fe-bf46-1740b74988c1" containerName="revoke-keys" Oct 29 23:37:06.672794 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.672592 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a971a987-be36-49fe-bf46-1740b74988c1" containerName="revoke-keys" Oct 29 23:37:06.672794 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.672662 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="a971a987-be36-49fe-bf46-1740b74988c1" containerName="revoke-keys" Oct 29 23:37:06.675884 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.675860 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.687936 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.687912 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-71f7db-openshift-default-dockercfg-fwm4w\"" Oct 29 23:37:06.709493 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.709448 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl"] Oct 29 23:37:06.772906 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.772868 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-podinfo\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.773099 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.772986 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-workload-certs\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.773099 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.773025 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-workload-socket\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.773099 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.773076 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-credential-socket\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.773281 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.773221 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c432c0b-430a-4beb-a88e-dbf7da35b599-istiod-ca-cert\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.773281 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.773262 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-token\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.773367 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.773337 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-envoy\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.773405 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.773374 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnhrk\" (UniqueName: \"kubernetes.io/projected/8c432c0b-430a-4beb-a88e-dbf7da35b599-kube-api-access-bnhrk\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.773449 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.773410 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-data\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.874174 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.874134 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-workload-socket\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.874358 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.874199 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-credential-socket\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.874358 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.874266 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c432c0b-430a-4beb-a88e-dbf7da35b599-istiod-ca-cert\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.874358 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.874299 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-token\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.874358 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.874327 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-envoy\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.874571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.874520 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnhrk\" (UniqueName: \"kubernetes.io/projected/8c432c0b-430a-4beb-a88e-dbf7da35b599-kube-api-access-bnhrk\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.874631 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.874590 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-data\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.874696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.874628 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-workload-socket\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.874696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.874636 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-podinfo\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.874798 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.874709 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-workload-certs\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.874798 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.874699 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-credential-socket\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.874931 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.874906 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-data\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.875078 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.875055 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-workload-certs\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.875078 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.875061 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c432c0b-430a-4beb-a88e-dbf7da35b599-istiod-ca-cert\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.876658 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.876631 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-envoy\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.877098 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.877074 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-podinfo\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.900221 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.900149 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnhrk\" (UniqueName: \"kubernetes.io/projected/8c432c0b-430a-4beb-a88e-dbf7da35b599-kube-api-access-bnhrk\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.902049 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.902019 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-token\") pod \"e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:06.993724 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:06.993634 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:07.303748 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:07.303727 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl"] Oct 29 23:37:07.306282 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:37:07.306253 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c432c0b_430a_4beb_a88e_dbf7da35b599.slice/crio-1f5de9ffe54971370ddd446fe0fea321209d0ccae25fb180d757441a3f787a07 WatchSource:0}: Error finding container 1f5de9ffe54971370ddd446fe0fea321209d0ccae25fb180d757441a3f787a07: Status 404 returned error can't find the container with id 1f5de9ffe54971370ddd446fe0fea321209d0ccae25fb180d757441a3f787a07 Oct 29 23:37:07.308496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:07.308433 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:37:07.308609 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:07.308551 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:37:07.308609 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:07.308599 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:37:08.260411 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:08.260365 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" event={"ID":"8c432c0b-430a-4beb-a88e-dbf7da35b599","Type":"ContainerStarted","Data":"7753779a331f1e99971a32232c69f3c072ced71c00756504217ba9c2bc58f69d"} Oct 29 23:37:08.260820 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:08.260436 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" event={"ID":"8c432c0b-430a-4beb-a88e-dbf7da35b599","Type":"ContainerStarted","Data":"1f5de9ffe54971370ddd446fe0fea321209d0ccae25fb180d757441a3f787a07"} Oct 29 23:37:08.306691 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:08.306630 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" podStartSLOduration=2.306616549 podStartE2EDuration="2.306616549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:08.305871303 +0000 UTC m=+1397.668876046" watchObservedRunningTime="2026-10-29 23:37:08.306616549 +0000 UTC m=+1397.669621342" Oct 29 23:37:08.994718 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:08.994673 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:08.999769 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:08.999743 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:09.265969 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:09.265882 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:09.266968 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:09.266951 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:12.660914 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.660879 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2"] Oct 29 23:37:12.664631 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.664612 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" Oct 29 23:37:12.685711 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.685681 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-71f7db\"" Oct 29 23:37:12.685711 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.685685 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-79e45fb8-dockercfg-bkmmf\"" Oct 29 23:37:12.688452 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.688427 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2"] Oct 29 23:37:12.743539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.743492 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7spdf\" (UniqueName: \"kubernetes.io/projected/58d2dfe9-b655-4198-97bb-7c31c0bb6597-kube-api-access-7spdf\") pod \"payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2\" (UID: \"58d2dfe9-b655-4198-97bb-7c31c0bb6597\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" Oct 29 23:37:12.743716 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.743642 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d2dfe9-b655-4198-97bb-7c31c0bb6597-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2\" (UID: \"58d2dfe9-b655-4198-97bb-7c31c0bb6597\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" Oct 29 23:37:12.774224 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.774174 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p"] Oct 29 23:37:12.779538 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.779498 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" Oct 29 23:37:12.799231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.799200 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p"] Oct 29 23:37:12.845271 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.845217 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d2dfe9-b655-4198-97bb-7c31c0bb6597-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2\" (UID: \"58d2dfe9-b655-4198-97bb-7c31c0bb6597\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" Oct 29 23:37:12.845582 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.845307 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7spdf\" (UniqueName: \"kubernetes.io/projected/58d2dfe9-b655-4198-97bb-7c31c0bb6597-kube-api-access-7spdf\") pod \"payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2\" (UID: \"58d2dfe9-b655-4198-97bb-7c31c0bb6597\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" Oct 29 23:37:12.846012 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.845986 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d2dfe9-b655-4198-97bb-7c31c0bb6597-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2\" (UID: \"58d2dfe9-b655-4198-97bb-7c31c0bb6597\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" Oct 29 23:37:12.863095 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.863067 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7spdf\" (UniqueName: \"kubernetes.io/projected/58d2dfe9-b655-4198-97bb-7c31c0bb6597-kube-api-access-7spdf\") pod \"payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2\" (UID: \"58d2dfe9-b655-4198-97bb-7c31c0bb6597\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" Oct 29 23:37:12.946133 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.946100 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/282f3fc6-4fe0-4a1e-8454-e98dc76c53b1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p\" (UID: \"282f3fc6-4fe0-4a1e-8454-e98dc76c53b1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" Oct 29 23:37:12.946307 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.946173 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkz5w\" (UniqueName: \"kubernetes.io/projected/282f3fc6-4fe0-4a1e-8454-e98dc76c53b1-kube-api-access-dkz5w\") pod \"payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p\" (UID: \"282f3fc6-4fe0-4a1e-8454-e98dc76c53b1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" Oct 29 23:37:12.978553 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:12.978515 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" Oct 29 23:37:13.047300 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:13.047267 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/282f3fc6-4fe0-4a1e-8454-e98dc76c53b1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p\" (UID: \"282f3fc6-4fe0-4a1e-8454-e98dc76c53b1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" Oct 29 23:37:13.047498 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:13.047352 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkz5w\" (UniqueName: \"kubernetes.io/projected/282f3fc6-4fe0-4a1e-8454-e98dc76c53b1-kube-api-access-dkz5w\") pod \"payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p\" (UID: \"282f3fc6-4fe0-4a1e-8454-e98dc76c53b1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" Oct 29 23:37:13.048606 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:13.048196 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/282f3fc6-4fe0-4a1e-8454-e98dc76c53b1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p\" (UID: \"282f3fc6-4fe0-4a1e-8454-e98dc76c53b1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" Oct 29 23:37:13.065111 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:13.065040 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkz5w\" (UniqueName: \"kubernetes.io/projected/282f3fc6-4fe0-4a1e-8454-e98dc76c53b1-kube-api-access-dkz5w\") pod \"payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p\" (UID: \"282f3fc6-4fe0-4a1e-8454-e98dc76c53b1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" Oct 29 23:37:13.096946 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:13.096913 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" Oct 29 23:37:13.278989 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:13.278883 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2"] Oct 29 23:37:13.283141 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:37:13.283113 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d2dfe9_b655_4198_97bb_7c31c0bb6597.slice/crio-3bea6a8a62586817ad14b36e4009f8bb81b9f6dfa24237ae00826f682227d6cb WatchSource:0}: Error finding container 3bea6a8a62586817ad14b36e4009f8bb81b9f6dfa24237ae00826f682227d6cb: Status 404 returned error can't find the container with id 3bea6a8a62586817ad14b36e4009f8bb81b9f6dfa24237ae00826f682227d6cb Oct 29 23:37:13.298597 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:13.298560 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" event={"ID":"58d2dfe9-b655-4198-97bb-7c31c0bb6597","Type":"ContainerStarted","Data":"3bea6a8a62586817ad14b36e4009f8bb81b9f6dfa24237ae00826f682227d6cb"} Oct 29 23:37:13.406578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:13.406545 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p"] Oct 29 23:37:13.410028 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:37:13.409996 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282f3fc6_4fe0_4a1e_8454_e98dc76c53b1.slice/crio-b7cf32bf037beaa900b7568ec732b45613521a788eb01291bf4d6d0e04705a40 WatchSource:0}: Error finding container b7cf32bf037beaa900b7568ec732b45613521a788eb01291bf4d6d0e04705a40: Status 404 returned error can't find the container with id b7cf32bf037beaa900b7568ec732b45613521a788eb01291bf4d6d0e04705a40 Oct 29 23:37:14.305906 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:14.305859 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" event={"ID":"282f3fc6-4fe0-4a1e-8454-e98dc76c53b1","Type":"ContainerStarted","Data":"f687e407362a1e4d62f211939e00fd7306f834744ba754eb31d7c6bd5eae6fcc"} Oct 29 23:37:14.306422 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:14.306048 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" Oct 29 23:37:14.306422 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:14.306100 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" event={"ID":"282f3fc6-4fe0-4a1e-8454-e98dc76c53b1","Type":"ContainerStarted","Data":"b7cf32bf037beaa900b7568ec732b45613521a788eb01291bf4d6d0e04705a40"} Oct 29 23:37:14.307738 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:14.307710 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" event={"ID":"58d2dfe9-b655-4198-97bb-7c31c0bb6597","Type":"ContainerStarted","Data":"29a6fb04806016a8e2c37975387781173e3565f73d697e08d0445cd2eb0527ad"} Oct 29 23:37:14.307845 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:14.307795 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" Oct 29 23:37:14.352942 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:14.350150 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" podStartSLOduration=2.350128963 podStartE2EDuration="2.350128963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:14.348848323 +0000 UTC m=+1403.711853047" watchObservedRunningTime="2026-10-29 23:37:14.350128963 +0000 UTC m=+1403.713133687" Oct 29 23:37:14.383452 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:14.383396 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" podStartSLOduration=2.383380852 podStartE2EDuration="2.383380852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:14.381604125 +0000 UTC m=+1403.744608848" watchObservedRunningTime="2026-10-29 23:37:14.383380852 +0000 UTC m=+1403.746385578" Oct 29 23:37:15.224866 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:37:15.224827 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:37:15.794485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:15.789421 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:37:25.305000 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:25.304696 6507 patch_prober.go:29] interesting pod/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:37:25.305000 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:25.304749 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" podUID="8c432c0b-430a-4beb-a88e-dbf7da35b599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:37:25.316173 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:25.316150 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" Oct 29 23:37:25.316303 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:25.316210 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" Oct 29 23:37:25.384449 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:25.384416 6507 generic.go:356] "Generic (PLEG): container finished" podID="8c432c0b-430a-4beb-a88e-dbf7da35b599" containerID="7753779a331f1e99971a32232c69f3c072ced71c00756504217ba9c2bc58f69d" exitCode=0 Oct 29 23:37:25.384643 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:25.384501 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" event={"ID":"8c432c0b-430a-4beb-a88e-dbf7da35b599","Type":"ContainerDied","Data":"7753779a331f1e99971a32232c69f3c072ced71c00756504217ba9c2bc58f69d"} Oct 29 23:37:25.384983 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:25.384967 6507 scope.go:122] "RemoveContainer" containerID="7753779a331f1e99971a32232c69f3c072ced71c00756504217ba9c2bc58f69d" Oct 29 23:37:25.386308 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:25.386277 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:37:25.386412 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:25.386359 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:37:25.386412 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:25.386401 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:37:26.392120 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:26.392085 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" event={"ID":"8c432c0b-430a-4beb-a88e-dbf7da35b599","Type":"ContainerStarted","Data":"6fd01fa276cfd6541857e4cc4d9787a12db0b9317674a021ebace58195d5faa3"} Oct 29 23:37:26.994846 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:26.994810 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:26.995066 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:26.994917 6507 patch_prober.go:29] interesting pod/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:37:26.995066 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:26.994965 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" podUID="8c432c0b-430a-4beb-a88e-dbf7da35b599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 29 23:37:27.994923 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:27.994890 6507 patch_prober.go:29] interesting pod/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:37:27.995336 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:27.994942 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" podUID="8c432c0b-430a-4beb-a88e-dbf7da35b599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 29 23:37:28.241150 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.241026 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw"] Oct 29 23:37:28.244918 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.244891 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.274196 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.274166 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-71f7db\"/\"kube-root-ca.crt\"" Oct 29 23:37:28.274355 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.274214 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-71f7db\"/\"openshift-service-ca.crt\"" Oct 29 23:37:28.274355 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.274310 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-71f7db\"/\"default-dockercfg-4sd7h\"" Oct 29 23:37:28.278821 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.278796 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-71f7db\"/\"simulator-single-main-71f7db-kserve-self-signed-certs\"" Oct 29 23:37:28.281022 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.280991 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw"] Oct 29 23:37:28.355957 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.355921 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-dshm\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.356143 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.355965 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-tmp-dir\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.356143 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.356054 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-tls-certs\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.356143 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.356110 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-home\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.356252 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.356227 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54sg9\" (UniqueName: \"kubernetes.io/projected/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-kube-api-access-54sg9\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.356290 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.356262 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-model-cache\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.456991 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.456952 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-model-cache\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.457176 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.457001 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-dshm\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.457176 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.457026 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-tmp-dir\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.457176 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.457051 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-tls-certs\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.457176 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.457085 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-home\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.457176 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.457162 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54sg9\" (UniqueName: \"kubernetes.io/projected/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-kube-api-access-54sg9\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.457473 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.457408 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-model-cache\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.457473 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.457417 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-tmp-dir\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.457705 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.457668 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-home\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.459445 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.459420 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-dshm\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.459630 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.459612 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-tls-certs\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.474791 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.474765 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54sg9\" (UniqueName: \"kubernetes.io/projected/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-kube-api-access-54sg9\") pod \"simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.560566 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.560478 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:28.887271 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.887199 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw"] Oct 29 23:37:28.890507 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:37:28.890482 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97d79d30_ffd2_4c9f_8a57_9875f4ec17a3.slice/crio-4e2c049d8384937d6c763eeab7a0b70daad50fa0456ac0b42fc4c18c4e30f84e WatchSource:0}: Error finding container 4e2c049d8384937d6c763eeab7a0b70daad50fa0456ac0b42fc4c18c4e30f84e: Status 404 returned error can't find the container with id 4e2c049d8384937d6c763eeab7a0b70daad50fa0456ac0b42fc4c18c4e30f84e Oct 29 23:37:28.994257 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.994163 6507 patch_prober.go:29] interesting pod/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:37:28.994257 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:28.994216 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" podUID="8c432c0b-430a-4beb-a88e-dbf7da35b599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 29 23:37:29.432074 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:29.432036 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" event={"ID":"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3","Type":"ContainerStarted","Data":"351e356584d2f2f1ae5e7960c8d38858676853f4d296457c317d24db7cc72bb1"} Oct 29 23:37:29.432595 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:29.432083 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" event={"ID":"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3","Type":"ContainerStarted","Data":"4e2c049d8384937d6c763eeab7a0b70daad50fa0456ac0b42fc4c18c4e30f84e"} Oct 29 23:37:29.468201 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:29.468137 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" podStartSLOduration=1.4681235990000001 podStartE2EDuration="1.468123599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:29.465585567 +0000 UTC m=+1418.828590286" watchObservedRunningTime="2026-10-29 23:37:29.468123599 +0000 UTC m=+1418.831128319" Oct 29 23:37:29.561327 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:29.561288 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:29.994945 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:29.994904 6507 patch_prober.go:29] interesting pod/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:37:29.995144 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:29.994960 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" podUID="8c432c0b-430a-4beb-a88e-dbf7da35b599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 29 23:37:30.481991 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:30.481959 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:37:30.993961 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:30.993924 6507 patch_prober.go:29] interesting pod/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 29 23:37:30.994147 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:30.993978 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" podUID="8c432c0b-430a-4beb-a88e-dbf7da35b599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 29 23:37:32.031658 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:32.030812 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:32.032077 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:32.031709 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:32.032695 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:32.032672 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:37:38.561747 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:38.561708 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:38.574653 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:38.574623 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:39.506539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:39.506512 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:37:42.025388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:42.025351 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2"] Oct 29 23:37:42.035333 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:42.035301 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-pd8t2"] Oct 29 23:37:42.751625 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:42.751591 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2711c384-96b8-4a4b-9e40-ab39da492c26" path="/var/lib/kubelet/pods/2711c384-96b8-4a4b-9e40-ab39da492c26/volumes" Oct 29 23:37:45.222876 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:37:45.222830 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:37:45.367010 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:45.366969 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:37:51.375937 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.375900 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96"] Oct 29 23:37:51.381292 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.381262 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" Oct 29 23:37:51.393434 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.393406 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:37:51.402999 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.402970 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:37:51.403128 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.403005 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:37:51.403128 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.403074 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:37:51.406376 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.406349 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96"] Oct 29 23:37:51.527896 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.527846 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf96c14-1212-49c5-98e8-babd7362dc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-881bcb76-9fs96\" (UID: \"adf96c14-1212-49c5-98e8-babd7362dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" Oct 29 23:37:51.528089 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.527916 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf96c14-1212-49c5-98e8-babd7362dc5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-881bcb76-9fs96\" (UID: \"adf96c14-1212-49c5-98e8-babd7362dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" Oct 29 23:37:51.528089 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.528011 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffzhp\" (UniqueName: \"kubernetes.io/projected/adf96c14-1212-49c5-98e8-babd7362dc5a-kube-api-access-ffzhp\") pod \"maas-api-revoke-sub-881bcb76-9fs96\" (UID: \"adf96c14-1212-49c5-98e8-babd7362dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" Oct 29 23:37:51.629615 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.629529 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf96c14-1212-49c5-98e8-babd7362dc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-881bcb76-9fs96\" (UID: \"adf96c14-1212-49c5-98e8-babd7362dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" Oct 29 23:37:51.629615 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.629599 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf96c14-1212-49c5-98e8-babd7362dc5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-881bcb76-9fs96\" (UID: \"adf96c14-1212-49c5-98e8-babd7362dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" Oct 29 23:37:51.629826 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.629631 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffzhp\" (UniqueName: \"kubernetes.io/projected/adf96c14-1212-49c5-98e8-babd7362dc5a-kube-api-access-ffzhp\") pod \"maas-api-revoke-sub-881bcb76-9fs96\" (UID: \"adf96c14-1212-49c5-98e8-babd7362dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" Oct 29 23:37:51.630189 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.630160 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf96c14-1212-49c5-98e8-babd7362dc5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-881bcb76-9fs96\" (UID: \"adf96c14-1212-49c5-98e8-babd7362dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" Oct 29 23:37:51.631927 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.631906 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf96c14-1212-49c5-98e8-babd7362dc5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-881bcb76-9fs96\" (UID: \"adf96c14-1212-49c5-98e8-babd7362dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" Oct 29 23:37:51.647734 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.647708 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffzhp\" (UniqueName: \"kubernetes.io/projected/adf96c14-1212-49c5-98e8-babd7362dc5a-kube-api-access-ffzhp\") pod \"maas-api-revoke-sub-881bcb76-9fs96\" (UID: \"adf96c14-1212-49c5-98e8-babd7362dc5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" Oct 29 23:37:51.695427 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.695393 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" Oct 29 23:37:51.988932 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:51.988897 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96"] Oct 29 23:37:51.992755 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:37:51.992723 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf96c14_1212_49c5_98e8_babd7362dc5a.slice/crio-10345e8cc3638da6eda3a508060f987d39a753d11e0b9453fac2bf33187f011b WatchSource:0}: Error finding container 10345e8cc3638da6eda3a508060f987d39a753d11e0b9453fac2bf33187f011b: Status 404 returned error can't find the container with id 10345e8cc3638da6eda3a508060f987d39a753d11e0b9453fac2bf33187f011b Oct 29 23:37:52.578572 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:52.578467 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" event={"ID":"adf96c14-1212-49c5-98e8-babd7362dc5a","Type":"ContainerStarted","Data":"0ce50e9bfb5ced8e1f8b31c9f5e77e05a9fc793ae1699ab77e90a7d7543881de"} Oct 29 23:37:52.578572 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:52.578534 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" event={"ID":"adf96c14-1212-49c5-98e8-babd7362dc5a","Type":"ContainerStarted","Data":"10345e8cc3638da6eda3a508060f987d39a753d11e0b9453fac2bf33187f011b"} Oct 29 23:37:52.616427 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:52.616373 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" podStartSLOduration=1.6163585459999998 podStartE2EDuration="1.616358546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:37:52.610051605 +0000 UTC m=+1441.973056327" watchObservedRunningTime="2026-10-29 23:37:52.616358546 +0000 UTC m=+1441.979363269" Oct 29 23:37:53.068278 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:53.068244 6507 scope.go:122] "RemoveContainer" containerID="468d95d1846e38b7841ff06d0804cd411e5c7a4f77ea26df2734233e77775c9c" Oct 29 23:37:53.076765 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:53.076731 6507 scope.go:122] "RemoveContainer" containerID="707ad2992e44631615d2558222c89eca368b466dc76a9034f9eb42118c2271c7" Oct 29 23:37:53.084660 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:53.084637 6507 scope.go:122] "RemoveContainer" containerID="2e6e5cdbc87158206f0119b424435c206ed920a6543f248c16d904857c075108" Oct 29 23:37:53.092537 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:53.092515 6507 scope.go:122] "RemoveContainer" containerID="545c963711ad13d69539d7488d754cda6476d444fba84e89d79cd8bc5976bd85" Oct 29 23:37:53.100597 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:53.100572 6507 scope.go:122] "RemoveContainer" containerID="e81cc6c99ad32489a654cfad34f098acf62fef155da8044fdfb617a7e45b0e7f" Oct 29 23:37:53.108911 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:53.108887 6507 scope.go:122] "RemoveContainer" containerID="4157c968937e4488e68d8d9174aa9009eb01fede7cd8e84c1d7a4f4ed66e557f" Oct 29 23:37:53.117182 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:53.117162 6507 scope.go:122] "RemoveContainer" containerID="9a96951b8483bcbf963b853b9bc292d022ce379f8fe82c0ffcc879704330ff8e" Oct 29 23:37:53.584960 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:53.584924 6507 generic.go:356] "Generic (PLEG): container finished" podID="adf96c14-1212-49c5-98e8-babd7362dc5a" containerID="0ce50e9bfb5ced8e1f8b31c9f5e77e05a9fc793ae1699ab77e90a7d7543881de" exitCode=0 Oct 29 23:37:53.585406 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:53.585007 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" event={"ID":"adf96c14-1212-49c5-98e8-babd7362dc5a","Type":"ContainerDied","Data":"0ce50e9bfb5ced8e1f8b31c9f5e77e05a9fc793ae1699ab77e90a7d7543881de"} Oct 29 23:37:54.725219 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:54.725194 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" Oct 29 23:37:54.880954 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:54.880860 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adf96c14-1212-49c5-98e8-babd7362dc5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf96c14-1212-49c5-98e8-babd7362dc5a-maas-api-xks-ca\") pod \"adf96c14-1212-49c5-98e8-babd7362dc5a\" (UID: \"adf96c14-1212-49c5-98e8-babd7362dc5a\") " Oct 29 23:37:54.880954 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:54.880916 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adf96c14-1212-49c5-98e8-babd7362dc5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf96c14-1212-49c5-98e8-babd7362dc5a-maas-api-openshift-ca\") pod \"adf96c14-1212-49c5-98e8-babd7362dc5a\" (UID: \"adf96c14-1212-49c5-98e8-babd7362dc5a\") " Oct 29 23:37:54.881219 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:54.880995 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf96c14-1212-49c5-98e8-babd7362dc5a-kube-api-access-ffzhp\" (UniqueName: \"kubernetes.io/projected/adf96c14-1212-49c5-98e8-babd7362dc5a-kube-api-access-ffzhp\") pod \"adf96c14-1212-49c5-98e8-babd7362dc5a\" (UID: \"adf96c14-1212-49c5-98e8-babd7362dc5a\") " Oct 29 23:37:54.881370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:54.881343 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf96c14-1212-49c5-98e8-babd7362dc5a-maas-api-openshift-ca" pod "adf96c14-1212-49c5-98e8-babd7362dc5a" (UID: "adf96c14-1212-49c5-98e8-babd7362dc5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:37:54.883175 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:54.883145 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf96c14-1212-49c5-98e8-babd7362dc5a-kube-api-access-ffzhp" pod "adf96c14-1212-49c5-98e8-babd7362dc5a" (UID: "adf96c14-1212-49c5-98e8-babd7362dc5a"). InnerVolumeSpecName "kube-api-access-ffzhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:37:54.883328 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:54.883312 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adf96c14-1212-49c5-98e8-babd7362dc5a-maas-api-xks-ca" pod "adf96c14-1212-49c5-98e8-babd7362dc5a" (UID: "adf96c14-1212-49c5-98e8-babd7362dc5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:37:54.982908 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:54.982869 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf96c14-1212-49c5-98e8-babd7362dc5a-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:37:54.982908 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:54.982903 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffzhp\" (UniqueName: \"kubernetes.io/projected/adf96c14-1212-49c5-98e8-babd7362dc5a-kube-api-access-ffzhp\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:37:54.983117 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:54.982919 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf96c14-1212-49c5-98e8-babd7362dc5a-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:37:55.597384 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:55.597352 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" Oct 29 23:37:55.597551 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:55.597384 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96" event={"ID":"adf96c14-1212-49c5-98e8-babd7362dc5a","Type":"ContainerDied","Data":"10345e8cc3638da6eda3a508060f987d39a753d11e0b9453fac2bf33187f011b"} Oct 29 23:37:55.597551 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:37:55.597427 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10345e8cc3638da6eda3a508060f987d39a753d11e0b9453fac2bf33187f011b" Oct 29 23:38:02.702623 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.702520 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:38:02.763319 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.763279 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk"] Oct 29 23:38:02.763982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.763960 6507 cpu_manager.go:414] "containerMap: removing container" podUID="adf96c14-1212-49c5-98e8-babd7362dc5a" containerName="revoke-keys" Oct 29 23:38:02.763982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.763983 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adf96c14-1212-49c5-98e8-babd7362dc5a" containerName="revoke-keys" Oct 29 23:38:02.764138 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.764111 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="adf96c14-1212-49c5-98e8-babd7362dc5a" containerName="revoke-keys" Oct 29 23:38:02.768384 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.768360 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk" Oct 29 23:38:02.780095 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.780074 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:38:02.783048 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.783027 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:38:02.792081 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.792055 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk"] Oct 29 23:38:02.792658 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.792636 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:38:02.792781 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.792688 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:38:02.877341 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.877302 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk\" (UID: \"6a3d20e0-9559-4404-9bbe-c083ab8bba3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk" Oct 29 23:38:02.877531 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.877376 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk\" (UID: \"6a3d20e0-9559-4404-9bbe-c083ab8bba3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk" Oct 29 23:38:02.877581 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.877544 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn4rf\" (UniqueName: \"kubernetes.io/projected/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-kube-api-access-dn4rf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk\" (UID: \"6a3d20e0-9559-4404-9bbe-c083ab8bba3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk" Oct 29 23:38:02.978838 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.978739 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn4rf\" (UniqueName: \"kubernetes.io/projected/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-kube-api-access-dn4rf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk\" (UID: \"6a3d20e0-9559-4404-9bbe-c083ab8bba3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk" Oct 29 23:38:02.978838 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.978784 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk\" (UID: \"6a3d20e0-9559-4404-9bbe-c083ab8bba3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk" Oct 29 23:38:02.978838 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.978834 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk\" (UID: \"6a3d20e0-9559-4404-9bbe-c083ab8bba3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk" Oct 29 23:38:02.979622 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.979596 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk\" (UID: \"6a3d20e0-9559-4404-9bbe-c083ab8bba3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk" Oct 29 23:38:02.981525 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.981500 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk\" (UID: \"6a3d20e0-9559-4404-9bbe-c083ab8bba3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk" Oct 29 23:38:02.995624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:02.995595 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn4rf\" (UniqueName: \"kubernetes.io/projected/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-kube-api-access-dn4rf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk\" (UID: \"6a3d20e0-9559-4404-9bbe-c083ab8bba3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk" Oct 29 23:38:03.082615 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:03.082570 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk" Oct 29 23:38:03.379537 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:03.379436 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk"] Oct 29 23:38:03.383506 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:38:03.383448 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3d20e0_9559_4404_9bbe_c083ab8bba3c.slice/crio-2be16651ce8e6d071d9138a62d02ab1907c66376ba4df9daa1ffd955fb16fe35 WatchSource:0}: Error finding container 2be16651ce8e6d071d9138a62d02ab1907c66376ba4df9daa1ffd955fb16fe35: Status 404 returned error can't find the container with id 2be16651ce8e6d071d9138a62d02ab1907c66376ba4df9daa1ffd955fb16fe35 Oct 29 23:38:03.640609 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:03.640526 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk" event={"ID":"6a3d20e0-9559-4404-9bbe-c083ab8bba3c","Type":"ContainerStarted","Data":"2be16651ce8e6d071d9138a62d02ab1907c66376ba4df9daa1ffd955fb16fe35"} Oct 29 23:38:04.667396 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:04.667302 6507 generic.go:356] "Generic (PLEG): container finished" podID="6a3d20e0-9559-4404-9bbe-c083ab8bba3c" containerID="6e7511ada42ece01b5bf49293cd17a60f24355a0289f7abd87f25a5ff939d550" exitCode=0 Oct 29 23:38:04.667839 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:04.667429 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk" event={"ID":"6a3d20e0-9559-4404-9bbe-c083ab8bba3c","Type":"ContainerDied","Data":"6e7511ada42ece01b5bf49293cd17a60f24355a0289f7abd87f25a5ff939d550"} Oct 29 23:38:05.817571 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:05.817543 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk" Oct 29 23:38:05.907144 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:05.907112 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-maas-api-xks-ca\") pod \"6a3d20e0-9559-4404-9bbe-c083ab8bba3c\" (UID: \"6a3d20e0-9559-4404-9bbe-c083ab8bba3c\") " Oct 29 23:38:05.907306 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:05.907164 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-kube-api-access-dn4rf\" (UniqueName: \"kubernetes.io/projected/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-kube-api-access-dn4rf\") pod \"6a3d20e0-9559-4404-9bbe-c083ab8bba3c\" (UID: \"6a3d20e0-9559-4404-9bbe-c083ab8bba3c\") " Oct 29 23:38:05.907306 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:05.907204 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-maas-api-openshift-ca\") pod \"6a3d20e0-9559-4404-9bbe-c083ab8bba3c\" (UID: \"6a3d20e0-9559-4404-9bbe-c083ab8bba3c\") " Oct 29 23:38:05.907641 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:05.907617 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-maas-api-openshift-ca" pod "6a3d20e0-9559-4404-9bbe-c083ab8bba3c" (UID: "6a3d20e0-9559-4404-9bbe-c083ab8bba3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:38:05.909324 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:05.909300 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-maas-api-xks-ca" pod "6a3d20e0-9559-4404-9bbe-c083ab8bba3c" (UID: "6a3d20e0-9559-4404-9bbe-c083ab8bba3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:38:05.909434 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:05.909371 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-kube-api-access-dn4rf" pod "6a3d20e0-9559-4404-9bbe-c083ab8bba3c" (UID: "6a3d20e0-9559-4404-9bbe-c083ab8bba3c"). InnerVolumeSpecName "kube-api-access-dn4rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:06.008976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:06.008942 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn4rf\" (UniqueName: \"kubernetes.io/projected/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-kube-api-access-dn4rf\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:06.008976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:06.008972 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:06.008976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:06.008982 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3d20e0-9559-4404-9bbe-c083ab8bba3c-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:06.679381 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:06.679353 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk" Oct 29 23:38:06.679565 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:06.679389 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk" event={"ID":"6a3d20e0-9559-4404-9bbe-c083ab8bba3c","Type":"ContainerDied","Data":"2be16651ce8e6d071d9138a62d02ab1907c66376ba4df9daa1ffd955fb16fe35"} Oct 29 23:38:06.679565 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:06.679440 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2be16651ce8e6d071d9138a62d02ab1907c66376ba4df9daa1ffd955fb16fe35" Oct 29 23:38:07.539958 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:07.539915 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw"] Oct 29 23:38:13.311776 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:13.311729 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p"] Oct 29 23:38:13.341957 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:13.341904 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2"] Oct 29 23:38:14.876214 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:14.876179 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk"] Oct 29 23:38:14.885299 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:14.885268 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-71f7db-5w2vk"] Oct 29 23:38:15.223702 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:38:15.223657 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:38:16.315833 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:16.315794 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:38:16.753147 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:16.753109 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a3d20e0-9559-4404-9bbe-c083ab8bba3c" path="/var/lib/kubelet/pods/6a3d20e0-9559-4404-9bbe-c083ab8bba3c/volumes" Oct 29 23:38:18.313103 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:18.313042 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" podUID="282f3fc6-4fe0-4a1e-8454-e98dc76c53b1" containerName="payload-processing" containerID="cri-o://f687e407362a1e4d62f211939e00fd7306f834744ba754eb31d7c6bd5eae6fcc" gracePeriod=25 Oct 29 23:38:18.342271 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:18.342231 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" podUID="58d2dfe9-b655-4198-97bb-7c31c0bb6597" containerName="payload-pre-processing" containerID="cri-o://29a6fb04806016a8e2c37975387781173e3565f73d697e08d0445cd2eb0527ad" gracePeriod=25 Oct 29 23:38:22.030774 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.030739 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl"] Oct 29 23:38:22.031254 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.031041 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" podUID="8c432c0b-430a-4beb-a88e-dbf7da35b599" containerName="istio-proxy" containerID="cri-o://6fd01fa276cfd6541857e4cc4d9787a12db0b9317674a021ebace58195d5faa3" gracePeriod=30 Oct 29 23:38:22.560010 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.559941 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" podUID="97d79d30-ffd2-4c9f-8a57-9875f4ec17a3" containerName="main" containerID="cri-o://351e356584d2f2f1ae5e7960c8d38858676853f4d296457c317d24db7cc72bb1" gracePeriod=45 Oct 29 23:38:22.777069 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.777037 6507 generic.go:356] "Generic (PLEG): container finished" podID="97d79d30-ffd2-4c9f-8a57-9875f4ec17a3" containerID="351e356584d2f2f1ae5e7960c8d38858676853f4d296457c317d24db7cc72bb1" exitCode=0 Oct 29 23:38:22.777278 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.777097 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" event={"ID":"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3","Type":"ContainerDied","Data":"351e356584d2f2f1ae5e7960c8d38858676853f4d296457c317d24db7cc72bb1"} Oct 29 23:38:22.812227 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.812160 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:38:22.984777 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.984736 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-tmp-dir\") pod \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " Oct 29 23:38:22.984978 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.984840 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-tls-certs\") pod \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " Oct 29 23:38:22.984978 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.984900 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-kube-api-access-54sg9\" (UniqueName: \"kubernetes.io/projected/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-kube-api-access-54sg9\") pod \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " Oct 29 23:38:22.984978 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.984965 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-dshm\") pod \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " Oct 29 23:38:22.985156 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.985019 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-home\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-home\") pod \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " Oct 29 23:38:22.985156 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.985047 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-tmp-dir" pod "97d79d30-ffd2-4c9f-8a57-9875f4ec17a3" (UID: "97d79d30-ffd2-4c9f-8a57-9875f4ec17a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:22.985156 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.985061 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-model-cache\") pod \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\" (UID: \"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3\") " Oct 29 23:38:22.985347 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.985310 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-home" pod "97d79d30-ffd2-4c9f-8a57-9875f4ec17a3" (UID: "97d79d30-ffd2-4c9f-8a57-9875f4ec17a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:22.985402 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.985339 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-model-cache" pod "97d79d30-ffd2-4c9f-8a57-9875f4ec17a3" (UID: "97d79d30-ffd2-4c9f-8a57-9875f4ec17a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:22.985567 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.985547 6507 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-model-cache\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:22.985639 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.985576 6507 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-tmp-dir\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:22.985639 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.985590 6507 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-home\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:22.987232 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.987208 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-dshm" pod "97d79d30-ffd2-4c9f-8a57-9875f4ec17a3" (UID: "97d79d30-ffd2-4c9f-8a57-9875f4ec17a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:22.987232 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.987221 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-tls-certs" pod "97d79d30-ffd2-4c9f-8a57-9875f4ec17a3" (UID: "97d79d30-ffd2-4c9f-8a57-9875f4ec17a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:38:22.987372 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:22.987264 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-kube-api-access-54sg9" pod "97d79d30-ffd2-4c9f-8a57-9875f4ec17a3" (UID: "97d79d30-ffd2-4c9f-8a57-9875f4ec17a3"). InnerVolumeSpecName "kube-api-access-54sg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:23.086720 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.086640 6507 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-tls-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:23.086720 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.086667 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54sg9\" (UniqueName: \"kubernetes.io/projected/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-kube-api-access-54sg9\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:23.086720 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.086677 6507 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3-dshm\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:23.296474 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.296425 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx"] Oct 29 23:38:23.305632 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.305607 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m7fzx"] Oct 29 23:38:23.352923 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.352848 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm"] Oct 29 23:38:23.353398 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.353379 6507 cpu_manager.go:414] "containerMap: removing container" podUID="6a3d20e0-9559-4404-9bbe-c083ab8bba3c" containerName="revoke-keys" Oct 29 23:38:23.353480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.353400 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3d20e0-9559-4404-9bbe-c083ab8bba3c" containerName="revoke-keys" Oct 29 23:38:23.353480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.353413 6507 cpu_manager.go:414] "containerMap: removing container" podUID="97d79d30-ffd2-4c9f-8a57-9875f4ec17a3" containerName="main" Oct 29 23:38:23.353480 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.353417 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97d79d30-ffd2-4c9f-8a57-9875f4ec17a3" containerName="main" Oct 29 23:38:23.353597 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.353500 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3d20e0-9559-4404-9bbe-c083ab8bba3c" containerName="revoke-keys" Oct 29 23:38:23.353597 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.353508 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="97d79d30-ffd2-4c9f-8a57-9875f4ec17a3" containerName="main" Oct 29 23:38:23.356879 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.356858 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" Oct 29 23:38:23.368958 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.368936 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:38:23.369061 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.368974 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:38:23.369061 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.368978 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:38:23.369061 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.369024 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:38:23.376291 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.376269 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm"] Oct 29 23:38:23.491909 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.491871 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d6pgm\" (UID: \"c3150fb0-7b66-4555-80dc-f3f5f9619f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" Oct 29 23:38:23.492088 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.491951 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgz7p\" (UniqueName: \"kubernetes.io/projected/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-kube-api-access-qgz7p\") pod \"maas-api-revoke-sub-3d4156f1-d6pgm\" (UID: \"c3150fb0-7b66-4555-80dc-f3f5f9619f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" Oct 29 23:38:23.492088 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.492054 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d6pgm\" (UID: \"c3150fb0-7b66-4555-80dc-f3f5f9619f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" Oct 29 23:38:23.593072 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.593027 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d6pgm\" (UID: \"c3150fb0-7b66-4555-80dc-f3f5f9619f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" Oct 29 23:38:23.593297 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.593094 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgz7p\" (UniqueName: \"kubernetes.io/projected/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-kube-api-access-qgz7p\") pod \"maas-api-revoke-sub-3d4156f1-d6pgm\" (UID: \"c3150fb0-7b66-4555-80dc-f3f5f9619f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" Oct 29 23:38:23.593297 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.593133 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d6pgm\" (UID: \"c3150fb0-7b66-4555-80dc-f3f5f9619f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" Oct 29 23:38:23.593805 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.593785 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d6pgm\" (UID: \"c3150fb0-7b66-4555-80dc-f3f5f9619f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" Oct 29 23:38:23.595586 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.595559 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d6pgm\" (UID: \"c3150fb0-7b66-4555-80dc-f3f5f9619f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" Oct 29 23:38:23.610414 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.610360 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgz7p\" (UniqueName: \"kubernetes.io/projected/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-kube-api-access-qgz7p\") pod \"maas-api-revoke-sub-3d4156f1-d6pgm\" (UID: \"c3150fb0-7b66-4555-80dc-f3f5f9619f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" Oct 29 23:38:23.669434 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.669393 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" Oct 29 23:38:23.783945 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.783912 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" Oct 29 23:38:23.784140 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.783943 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw" event={"ID":"97d79d30-ffd2-4c9f-8a57-9875f4ec17a3","Type":"ContainerDied","Data":"4e2c049d8384937d6c763eeab7a0b70daad50fa0456ac0b42fc4c18c4e30f84e"} Oct 29 23:38:23.784140 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.783993 6507 scope.go:122] "RemoveContainer" containerID="351e356584d2f2f1ae5e7960c8d38858676853f4d296457c317d24db7cc72bb1" Oct 29 23:38:23.827002 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.826973 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw"] Oct 29 23:38:23.830619 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.830591 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-71f7db/simulator-single-main-71f7db-kserve-6c7cfc447-nf6tw"] Oct 29 23:38:23.965512 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:23.965480 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm"] Oct 29 23:38:23.968883 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:38:23.968852 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3150fb0_7b66_4555_80dc_f3f5f9619f7e.slice/crio-af65abb497c344895066cd0412caf3d0adb81e23001b645813bb2a6a0931438a WatchSource:0}: Error finding container af65abb497c344895066cd0412caf3d0adb81e23001b645813bb2a6a0931438a: Status 404 returned error can't find the container with id af65abb497c344895066cd0412caf3d0adb81e23001b645813bb2a6a0931438a Oct 29 23:38:24.752041 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:24.752007 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97d79d30-ffd2-4c9f-8a57-9875f4ec17a3" path="/var/lib/kubelet/pods/97d79d30-ffd2-4c9f-8a57-9875f4ec17a3/volumes" Oct 29 23:38:24.752447 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:24.752432 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a971a987-be36-49fe-bf46-1740b74988c1" path="/var/lib/kubelet/pods/a971a987-be36-49fe-bf46-1740b74988c1/volumes" Oct 29 23:38:24.791816 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:24.791785 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" event={"ID":"c3150fb0-7b66-4555-80dc-f3f5f9619f7e","Type":"ContainerStarted","Data":"9312838268e0c81f4fff3c18a9978134384dff9e9540b5400f04bfba5ca8d97a"} Oct 29 23:38:24.791990 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:24.791847 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" event={"ID":"c3150fb0-7b66-4555-80dc-f3f5f9619f7e","Type":"ContainerStarted","Data":"af65abb497c344895066cd0412caf3d0adb81e23001b645813bb2a6a0931438a"} Oct 29 23:38:24.842604 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:24.842557 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" podStartSLOduration=1.8425445489999999 podStartE2EDuration="1.842544549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:38:24.842341638 +0000 UTC m=+1474.205346360" watchObservedRunningTime="2026-10-29 23:38:24.842544549 +0000 UTC m=+1474.205549272" Oct 29 23:38:25.315307 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:25.315265 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" podUID="58d2dfe9-b655-4198-97bb-7c31c0bb6597" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:38:25.315511 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:25.315273 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" podUID="282f3fc6-4fe0-4a1e-8454-e98dc76c53b1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:38:25.799611 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:25.799571 6507 generic.go:356] "Generic (PLEG): container finished" podID="c3150fb0-7b66-4555-80dc-f3f5f9619f7e" containerID="9312838268e0c81f4fff3c18a9978134384dff9e9540b5400f04bfba5ca8d97a" exitCode=0 Oct 29 23:38:25.800061 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:25.799662 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" event={"ID":"c3150fb0-7b66-4555-80dc-f3f5f9619f7e","Type":"ContainerDied","Data":"9312838268e0c81f4fff3c18a9978134384dff9e9540b5400f04bfba5ca8d97a"} Oct 29 23:38:26.943296 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:26.943269 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" Oct 29 23:38:27.129937 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.129859 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-maas-api-xks-ca\") pod \"c3150fb0-7b66-4555-80dc-f3f5f9619f7e\" (UID: \"c3150fb0-7b66-4555-80dc-f3f5f9619f7e\") " Oct 29 23:38:27.129937 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.129914 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-maas-api-openshift-ca\") pod \"c3150fb0-7b66-4555-80dc-f3f5f9619f7e\" (UID: \"c3150fb0-7b66-4555-80dc-f3f5f9619f7e\") " Oct 29 23:38:27.130146 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.129969 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-kube-api-access-qgz7p\" (UniqueName: \"kubernetes.io/projected/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-kube-api-access-qgz7p\") pod \"c3150fb0-7b66-4555-80dc-f3f5f9619f7e\" (UID: \"c3150fb0-7b66-4555-80dc-f3f5f9619f7e\") " Oct 29 23:38:27.130277 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.130252 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-maas-api-openshift-ca" pod "c3150fb0-7b66-4555-80dc-f3f5f9619f7e" (UID: "c3150fb0-7b66-4555-80dc-f3f5f9619f7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:38:27.130370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.130356 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:27.132094 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.132064 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-maas-api-xks-ca" pod "c3150fb0-7b66-4555-80dc-f3f5f9619f7e" (UID: "c3150fb0-7b66-4555-80dc-f3f5f9619f7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:38:27.132202 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.132113 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-kube-api-access-qgz7p" pod "c3150fb0-7b66-4555-80dc-f3f5f9619f7e" (UID: "c3150fb0-7b66-4555-80dc-f3f5f9619f7e"). InnerVolumeSpecName "kube-api-access-qgz7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:27.231345 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.231306 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgz7p\" (UniqueName: \"kubernetes.io/projected/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-kube-api-access-qgz7p\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:27.231514 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.231349 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3150fb0-7b66-4555-80dc-f3f5f9619f7e-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:27.271761 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.271734 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:38:27.433878 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.433791 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-envoy\") pod \"8c432c0b-430a-4beb-a88e-dbf7da35b599\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " Oct 29 23:38:27.433878 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.433872 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c432c0b-430a-4beb-a88e-dbf7da35b599-kube-api-access-bnhrk\" (UniqueName: \"kubernetes.io/projected/8c432c0b-430a-4beb-a88e-dbf7da35b599-kube-api-access-bnhrk\") pod \"8c432c0b-430a-4beb-a88e-dbf7da35b599\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " Oct 29 23:38:27.434074 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.433945 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-podinfo\") pod \"8c432c0b-430a-4beb-a88e-dbf7da35b599\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " Oct 29 23:38:27.434074 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.434000 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-credential-socket\") pod \"8c432c0b-430a-4beb-a88e-dbf7da35b599\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " Oct 29 23:38:27.434074 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.434028 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-workload-certs\") pod \"8c432c0b-430a-4beb-a88e-dbf7da35b599\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " Oct 29 23:38:27.434232 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.434070 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-workload-socket\") pod \"8c432c0b-430a-4beb-a88e-dbf7da35b599\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " Oct 29 23:38:27.434232 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.434113 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c432c0b-430a-4beb-a88e-dbf7da35b599-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c432c0b-430a-4beb-a88e-dbf7da35b599-istiod-ca-cert\") pod \"8c432c0b-430a-4beb-a88e-dbf7da35b599\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " Oct 29 23:38:27.434668 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.434221 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-token\" (UniqueName: \"kubernetes.io/projected/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-token\") pod \"8c432c0b-430a-4beb-a88e-dbf7da35b599\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " Oct 29 23:38:27.434668 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.434313 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-data\") pod \"8c432c0b-430a-4beb-a88e-dbf7da35b599\" (UID: \"8c432c0b-430a-4beb-a88e-dbf7da35b599\") " Oct 29 23:38:27.434668 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.434321 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-credential-socket" pod "8c432c0b-430a-4beb-a88e-dbf7da35b599" (UID: "8c432c0b-430a-4beb-a88e-dbf7da35b599"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:27.434668 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.434449 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-workload-certs" pod "8c432c0b-430a-4beb-a88e-dbf7da35b599" (UID: "8c432c0b-430a-4beb-a88e-dbf7da35b599"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:27.434668 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.434554 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-workload-socket" pod "8c432c0b-430a-4beb-a88e-dbf7da35b599" (UID: "8c432c0b-430a-4beb-a88e-dbf7da35b599"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:27.434944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.434677 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-data" pod "8c432c0b-430a-4beb-a88e-dbf7da35b599" (UID: "8c432c0b-430a-4beb-a88e-dbf7da35b599"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:27.434944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.434736 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c432c0b-430a-4beb-a88e-dbf7da35b599-istiod-ca-cert" pod "8c432c0b-430a-4beb-a88e-dbf7da35b599" (UID: "8c432c0b-430a-4beb-a88e-dbf7da35b599"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:38:27.434944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.434797 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:27.434944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.434813 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:27.434944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.434827 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:27.434944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.434839 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:27.436179 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.436159 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-podinfo" pod "8c432c0b-430a-4beb-a88e-dbf7da35b599" (UID: "8c432c0b-430a-4beb-a88e-dbf7da35b599"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:38:27.436400 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.436383 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-token" pod "8c432c0b-430a-4beb-a88e-dbf7da35b599" (UID: "8c432c0b-430a-4beb-a88e-dbf7da35b599"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:27.436570 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.436546 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-envoy" pod "8c432c0b-430a-4beb-a88e-dbf7da35b599" (UID: "8c432c0b-430a-4beb-a88e-dbf7da35b599"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:38:27.436570 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.436562 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c432c0b-430a-4beb-a88e-dbf7da35b599-kube-api-access-bnhrk" pod "8c432c0b-430a-4beb-a88e-dbf7da35b599" (UID: "8c432c0b-430a-4beb-a88e-dbf7da35b599"). InnerVolumeSpecName "kube-api-access-bnhrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:27.536165 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.536130 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c432c0b-430a-4beb-a88e-dbf7da35b599-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:27.536165 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.536160 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:27.536165 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.536168 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:27.536388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.536177 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnhrk\" (UniqueName: \"kubernetes.io/projected/8c432c0b-430a-4beb-a88e-dbf7da35b599-kube-api-access-bnhrk\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:27.536388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.536185 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c432c0b-430a-4beb-a88e-dbf7da35b599-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:27.811510 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.811476 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" Oct 29 23:38:27.811711 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.811476 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm" event={"ID":"c3150fb0-7b66-4555-80dc-f3f5f9619f7e","Type":"ContainerDied","Data":"af65abb497c344895066cd0412caf3d0adb81e23001b645813bb2a6a0931438a"} Oct 29 23:38:27.811711 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.811616 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af65abb497c344895066cd0412caf3d0adb81e23001b645813bb2a6a0931438a" Oct 29 23:38:27.813251 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.813226 6507 generic.go:356] "Generic (PLEG): container finished" podID="8c432c0b-430a-4beb-a88e-dbf7da35b599" containerID="6fd01fa276cfd6541857e4cc4d9787a12db0b9317674a021ebace58195d5faa3" exitCode=0 Oct 29 23:38:27.813402 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.813271 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" Oct 29 23:38:27.813402 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.813313 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" event={"ID":"8c432c0b-430a-4beb-a88e-dbf7da35b599","Type":"ContainerDied","Data":"6fd01fa276cfd6541857e4cc4d9787a12db0b9317674a021ebace58195d5faa3"} Oct 29 23:38:27.813402 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.813359 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl" event={"ID":"8c432c0b-430a-4beb-a88e-dbf7da35b599","Type":"ContainerDied","Data":"1f5de9ffe54971370ddd446fe0fea321209d0ccae25fb180d757441a3f787a07"} Oct 29 23:38:27.813402 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.813382 6507 scope.go:122] "RemoveContainer" containerID="6fd01fa276cfd6541857e4cc4d9787a12db0b9317674a021ebace58195d5faa3" Oct 29 23:38:27.823380 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.823365 6507 scope.go:122] "RemoveContainer" containerID="7753779a331f1e99971a32232c69f3c072ced71c00756504217ba9c2bc58f69d" Oct 29 23:38:27.832070 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.832055 6507 scope.go:122] "RemoveContainer" containerID="6fd01fa276cfd6541857e4cc4d9787a12db0b9317674a021ebace58195d5faa3" Oct 29 23:38:27.832325 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:38:27.832305 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd01fa276cfd6541857e4cc4d9787a12db0b9317674a021ebace58195d5faa3\": container with ID starting with 6fd01fa276cfd6541857e4cc4d9787a12db0b9317674a021ebace58195d5faa3 not found: ID does not exist" containerID="6fd01fa276cfd6541857e4cc4d9787a12db0b9317674a021ebace58195d5faa3" Oct 29 23:38:27.832378 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.832337 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd01fa276cfd6541857e4cc4d9787a12db0b9317674a021ebace58195d5faa3"} err="failed to get container status \"6fd01fa276cfd6541857e4cc4d9787a12db0b9317674a021ebace58195d5faa3\": rpc error: code = NotFound desc = could not find container \"6fd01fa276cfd6541857e4cc4d9787a12db0b9317674a021ebace58195d5faa3\": container with ID starting with 6fd01fa276cfd6541857e4cc4d9787a12db0b9317674a021ebace58195d5faa3 not found: ID does not exist" Oct 29 23:38:27.832378 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.832358 6507 scope.go:122] "RemoveContainer" containerID="7753779a331f1e99971a32232c69f3c072ced71c00756504217ba9c2bc58f69d" Oct 29 23:38:27.832652 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:38:27.832624 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7753779a331f1e99971a32232c69f3c072ced71c00756504217ba9c2bc58f69d\": container with ID starting with 7753779a331f1e99971a32232c69f3c072ced71c00756504217ba9c2bc58f69d not found: ID does not exist" containerID="7753779a331f1e99971a32232c69f3c072ced71c00756504217ba9c2bc58f69d" Oct 29 23:38:27.832726 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.832662 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7753779a331f1e99971a32232c69f3c072ced71c00756504217ba9c2bc58f69d"} err="failed to get container status \"7753779a331f1e99971a32232c69f3c072ced71c00756504217ba9c2bc58f69d\": rpc error: code = NotFound desc = could not find container \"7753779a331f1e99971a32232c69f3c072ced71c00756504217ba9c2bc58f69d\": container with ID starting with 7753779a331f1e99971a32232c69f3c072ced71c00756504217ba9c2bc58f69d not found: ID does not exist" Oct 29 23:38:27.858356 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.858326 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl"] Oct 29 23:38:27.867399 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:27.867374 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-71f7db-openshift-default-7b85d484f-sc8gl"] Oct 29 23:38:28.751717 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:28.751684 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c432c0b-430a-4beb-a88e-dbf7da35b599" path="/var/lib/kubelet/pods/8c432c0b-430a-4beb-a88e-dbf7da35b599/volumes" Oct 29 23:38:31.026484 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:31.026411 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p"] Oct 29 23:38:31.035635 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:31.035590 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-t4p6p"] Oct 29 23:38:32.752008 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:32.751975 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c109ab0d-5a84-45d3-bb8f-9e29a34d416a" path="/var/lib/kubelet/pods/c109ab0d-5a84-45d3-bb8f-9e29a34d416a/volumes" Oct 29 23:38:35.314774 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:35.314729 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" podUID="282f3fc6-4fe0-4a1e-8454-e98dc76c53b1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:38:35.314774 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:35.314764 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" podUID="58d2dfe9-b655-4198-97bb-7c31c0bb6597" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:38:38.596755 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.596724 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" Oct 29 23:38:38.625179 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.625153 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" Oct 29 23:38:38.638002 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.637964 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/282f3fc6-4fe0-4a1e-8454-e98dc76c53b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/282f3fc6-4fe0-4a1e-8454-e98dc76c53b1-plugins-config-volume\") pod \"282f3fc6-4fe0-4a1e-8454-e98dc76c53b1\" (UID: \"282f3fc6-4fe0-4a1e-8454-e98dc76c53b1\") " Oct 29 23:38:38.638174 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.638060 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/282f3fc6-4fe0-4a1e-8454-e98dc76c53b1-kube-api-access-dkz5w\" (UniqueName: \"kubernetes.io/projected/282f3fc6-4fe0-4a1e-8454-e98dc76c53b1-kube-api-access-dkz5w\") pod \"282f3fc6-4fe0-4a1e-8454-e98dc76c53b1\" (UID: \"282f3fc6-4fe0-4a1e-8454-e98dc76c53b1\") " Oct 29 23:38:38.638400 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.638372 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/282f3fc6-4fe0-4a1e-8454-e98dc76c53b1-plugins-config-volume" pod "282f3fc6-4fe0-4a1e-8454-e98dc76c53b1" (UID: "282f3fc6-4fe0-4a1e-8454-e98dc76c53b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:38:38.638578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.638559 6507 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/282f3fc6-4fe0-4a1e-8454-e98dc76c53b1-plugins-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:38.640328 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.640306 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282f3fc6-4fe0-4a1e-8454-e98dc76c53b1-kube-api-access-dkz5w" pod "282f3fc6-4fe0-4a1e-8454-e98dc76c53b1" (UID: "282f3fc6-4fe0-4a1e-8454-e98dc76c53b1"). InnerVolumeSpecName "kube-api-access-dkz5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:38.739792 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.739763 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d2dfe9-b655-4198-97bb-7c31c0bb6597-kube-api-access-7spdf\" (UniqueName: \"kubernetes.io/projected/58d2dfe9-b655-4198-97bb-7c31c0bb6597-kube-api-access-7spdf\") pod \"58d2dfe9-b655-4198-97bb-7c31c0bb6597\" (UID: \"58d2dfe9-b655-4198-97bb-7c31c0bb6597\") " Oct 29 23:38:38.739975 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.739826 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58d2dfe9-b655-4198-97bb-7c31c0bb6597-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d2dfe9-b655-4198-97bb-7c31c0bb6597-plugins-config-volume\") pod \"58d2dfe9-b655-4198-97bb-7c31c0bb6597\" (UID: \"58d2dfe9-b655-4198-97bb-7c31c0bb6597\") " Oct 29 23:38:38.740089 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.740072 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkz5w\" (UniqueName: \"kubernetes.io/projected/282f3fc6-4fe0-4a1e-8454-e98dc76c53b1-kube-api-access-dkz5w\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:38.740232 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.740207 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d2dfe9-b655-4198-97bb-7c31c0bb6597-plugins-config-volume" pod "58d2dfe9-b655-4198-97bb-7c31c0bb6597" (UID: "58d2dfe9-b655-4198-97bb-7c31c0bb6597"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:38:38.741987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.741963 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d2dfe9-b655-4198-97bb-7c31c0bb6597-kube-api-access-7spdf" pod "58d2dfe9-b655-4198-97bb-7c31c0bb6597" (UID: "58d2dfe9-b655-4198-97bb-7c31c0bb6597"). InnerVolumeSpecName "kube-api-access-7spdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:38:38.840790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.840699 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7spdf\" (UniqueName: \"kubernetes.io/projected/58d2dfe9-b655-4198-97bb-7c31c0bb6597-kube-api-access-7spdf\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:38.840790 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.840734 6507 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58d2dfe9-b655-4198-97bb-7c31c0bb6597-plugins-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:38:38.871683 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.871646 6507 generic.go:356] "Generic (PLEG): container finished" podID="282f3fc6-4fe0-4a1e-8454-e98dc76c53b1" containerID="f687e407362a1e4d62f211939e00fd7306f834744ba754eb31d7c6bd5eae6fcc" exitCode=0 Oct 29 23:38:38.871865 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.871696 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" Oct 29 23:38:38.871865 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.871730 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" event={"ID":"282f3fc6-4fe0-4a1e-8454-e98dc76c53b1","Type":"ContainerDied","Data":"f687e407362a1e4d62f211939e00fd7306f834744ba754eb31d7c6bd5eae6fcc"} Oct 29 23:38:38.871865 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.871790 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p" event={"ID":"282f3fc6-4fe0-4a1e-8454-e98dc76c53b1","Type":"ContainerDied","Data":"b7cf32bf037beaa900b7568ec732b45613521a788eb01291bf4d6d0e04705a40"} Oct 29 23:38:38.871865 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.871822 6507 scope.go:122] "RemoveContainer" containerID="f687e407362a1e4d62f211939e00fd7306f834744ba754eb31d7c6bd5eae6fcc" Oct 29 23:38:38.873146 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.873121 6507 generic.go:356] "Generic (PLEG): container finished" podID="58d2dfe9-b655-4198-97bb-7c31c0bb6597" containerID="29a6fb04806016a8e2c37975387781173e3565f73d697e08d0445cd2eb0527ad" exitCode=0 Oct 29 23:38:38.873269 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.873148 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" event={"ID":"58d2dfe9-b655-4198-97bb-7c31c0bb6597","Type":"ContainerDied","Data":"29a6fb04806016a8e2c37975387781173e3565f73d697e08d0445cd2eb0527ad"} Oct 29 23:38:38.873269 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.873173 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" Oct 29 23:38:38.873269 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.873190 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2" event={"ID":"58d2dfe9-b655-4198-97bb-7c31c0bb6597","Type":"ContainerDied","Data":"3bea6a8a62586817ad14b36e4009f8bb81b9f6dfa24237ae00826f682227d6cb"} Oct 29 23:38:38.880524 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.880506 6507 scope.go:122] "RemoveContainer" containerID="f687e407362a1e4d62f211939e00fd7306f834744ba754eb31d7c6bd5eae6fcc" Oct 29 23:38:38.880778 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:38:38.880757 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f687e407362a1e4d62f211939e00fd7306f834744ba754eb31d7c6bd5eae6fcc\": container with ID starting with f687e407362a1e4d62f211939e00fd7306f834744ba754eb31d7c6bd5eae6fcc not found: ID does not exist" containerID="f687e407362a1e4d62f211939e00fd7306f834744ba754eb31d7c6bd5eae6fcc" Oct 29 23:38:38.880827 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.880786 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f687e407362a1e4d62f211939e00fd7306f834744ba754eb31d7c6bd5eae6fcc"} err="failed to get container status \"f687e407362a1e4d62f211939e00fd7306f834744ba754eb31d7c6bd5eae6fcc\": rpc error: code = NotFound desc = could not find container \"f687e407362a1e4d62f211939e00fd7306f834744ba754eb31d7c6bd5eae6fcc\": container with ID starting with f687e407362a1e4d62f211939e00fd7306f834744ba754eb31d7c6bd5eae6fcc not found: ID does not exist" Oct 29 23:38:38.880827 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.880802 6507 scope.go:122] "RemoveContainer" containerID="29a6fb04806016a8e2c37975387781173e3565f73d697e08d0445cd2eb0527ad" Oct 29 23:38:38.888886 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.888863 6507 scope.go:122] "RemoveContainer" containerID="29a6fb04806016a8e2c37975387781173e3565f73d697e08d0445cd2eb0527ad" Oct 29 23:38:38.889158 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:38:38.889139 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a6fb04806016a8e2c37975387781173e3565f73d697e08d0445cd2eb0527ad\": container with ID starting with 29a6fb04806016a8e2c37975387781173e3565f73d697e08d0445cd2eb0527ad not found: ID does not exist" containerID="29a6fb04806016a8e2c37975387781173e3565f73d697e08d0445cd2eb0527ad" Oct 29 23:38:38.889218 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.889166 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a6fb04806016a8e2c37975387781173e3565f73d697e08d0445cd2eb0527ad"} err="failed to get container status \"29a6fb04806016a8e2c37975387781173e3565f73d697e08d0445cd2eb0527ad\": rpc error: code = NotFound desc = could not find container \"29a6fb04806016a8e2c37975387781173e3565f73d697e08d0445cd2eb0527ad\": container with ID starting with 29a6fb04806016a8e2c37975387781173e3565f73d697e08d0445cd2eb0527ad not found: ID does not exist" Oct 29 23:38:38.914363 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.914330 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2"] Oct 29 23:38:38.923442 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.923410 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-71f7db-5f4454cqp2"] Oct 29 23:38:38.945834 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.945800 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p"] Oct 29 23:38:38.949216 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:38.949190 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-71f7db-748867594ntz7p"] Oct 29 23:38:40.751356 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:40.751324 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="282f3fc6-4fe0-4a1e-8454-e98dc76c53b1" path="/var/lib/kubelet/pods/282f3fc6-4fe0-4a1e-8454-e98dc76c53b1/volumes" Oct 29 23:38:40.751753 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:40.751713 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d2dfe9-b655-4198-97bb-7c31c0bb6597" path="/var/lib/kubelet/pods/58d2dfe9-b655-4198-97bb-7c31c0bb6597/volumes" Oct 29 23:38:45.222806 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:38:45.222768 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:38:52.026237 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:52.026202 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9"] Oct 29 23:38:52.035633 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:52.035600 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-znjl9"] Oct 29 23:38:52.751768 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:52.751734 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99172142-8d97-4b7b-87cb-9e1f8349633e" path="/var/lib/kubelet/pods/99172142-8d97-4b7b-87cb-9e1f8349633e/volumes" Oct 29 23:38:53.159201 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:38:53.159112 6507 scope.go:122] "RemoveContainer" containerID="9ac5a40ba91772ef032d84e9ad998e026bcf0043f4e5db769debe97767a64ef8" Oct 29 23:39:03.024925 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:39:03.024894 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq"] Oct 29 23:39:03.034401 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:39:03.034370 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bxnvq"] Oct 29 23:39:04.751178 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:39:04.751144 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a4dc6f-8533-4917-8da8-4902ebec72d1" path="/var/lib/kubelet/pods/09a4dc6f-8533-4917-8da8-4902ebec72d1/volumes" Oct 29 23:39:15.222769 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:39:15.222726 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:39:31.033238 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:39:31.033157 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td"] Oct 29 23:39:31.035432 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:39:31.035404 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xz2td"] Oct 29 23:39:32.751696 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:39:32.751664 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0de3c32-2943-469d-ac61-b6377f30c220" path="/var/lib/kubelet/pods/a0de3c32-2943-469d-ac61-b6377f30c220/volumes" Oct 29 23:39:42.028147 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:39:42.028113 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn"] Oct 29 23:39:42.036746 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:39:42.036721 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5zltn"] Oct 29 23:39:42.751680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:39:42.751645 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847c16f6-eacf-4f57-87f8-9dbaddbcf884" path="/var/lib/kubelet/pods/847c16f6-eacf-4f57-87f8-9dbaddbcf884/volumes" Oct 29 23:39:45.224237 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:39:45.224197 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:39:53.195131 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:39:53.195095 6507 scope.go:122] "RemoveContainer" containerID="d5380f2d8c08083f65761629476755517137161e334d1854b85b36c7cc8b7497" Oct 29 23:39:53.204035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:39:53.204007 6507 scope.go:122] "RemoveContainer" containerID="521991e634d099d910846f01bb698ebaf0772e65abf02a5f7a269234eccb4942" Oct 29 23:40:03.024476 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:03.024436 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24"] Oct 29 23:40:03.033472 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:03.033437 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9zb24"] Oct 29 23:40:03.979954 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:03.979917 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:40:04.754905 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:04.754858 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7c73b24-9b5d-4883-9873-8eb1be6a6b68" path="/var/lib/kubelet/pods/e7c73b24-9b5d-4883-9873-8eb1be6a6b68/volumes" Oct 29 23:40:08.029675 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.029637 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt"] Oct 29 23:40:08.030248 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.030214 6507 cpu_manager.go:414] "containerMap: removing container" podUID="8c432c0b-430a-4beb-a88e-dbf7da35b599" containerName="istio-proxy" Oct 29 23:40:08.030248 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.030248 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c432c0b-430a-4beb-a88e-dbf7da35b599" containerName="istio-proxy" Oct 29 23:40:08.030388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.030261 6507 cpu_manager.go:414] "containerMap: removing container" podUID="282f3fc6-4fe0-4a1e-8454-e98dc76c53b1" containerName="payload-processing" Oct 29 23:40:08.030388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.030266 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="282f3fc6-4fe0-4a1e-8454-e98dc76c53b1" containerName="payload-processing" Oct 29 23:40:08.030388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.030282 6507 cpu_manager.go:414] "containerMap: removing container" podUID="58d2dfe9-b655-4198-97bb-7c31c0bb6597" containerName="payload-pre-processing" Oct 29 23:40:08.030388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.030292 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d2dfe9-b655-4198-97bb-7c31c0bb6597" containerName="payload-pre-processing" Oct 29 23:40:08.030388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.030322 6507 cpu_manager.go:414] "containerMap: removing container" podUID="c3150fb0-7b66-4555-80dc-f3f5f9619f7e" containerName="revoke-keys" Oct 29 23:40:08.030388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.030327 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3150fb0-7b66-4555-80dc-f3f5f9619f7e" containerName="revoke-keys" Oct 29 23:40:08.030388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.030386 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3150fb0-7b66-4555-80dc-f3f5f9619f7e" containerName="revoke-keys" Oct 29 23:40:08.030759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.030394 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c432c0b-430a-4beb-a88e-dbf7da35b599" containerName="istio-proxy" Oct 29 23:40:08.030759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.030400 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="282f3fc6-4fe0-4a1e-8454-e98dc76c53b1" containerName="payload-processing" Oct 29 23:40:08.030759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.030408 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d2dfe9-b655-4198-97bb-7c31c0bb6597" containerName="payload-pre-processing" Oct 29 23:40:08.030759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.030418 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c432c0b-430a-4beb-a88e-dbf7da35b599" containerName="istio-proxy" Oct 29 23:40:08.035065 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.035048 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" Oct 29 23:40:08.037772 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.037742 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvwsg\" (UniqueName: \"kubernetes.io/projected/11b18d64-a832-43f3-9dd7-a33f06a96422-kube-api-access-lvwsg\") pod \"maas-api-revoke-sub-d576e68b-bvjwt\" (UID: \"11b18d64-a832-43f3-9dd7-a33f06a96422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" Oct 29 23:40:08.037941 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.037919 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b18d64-a832-43f3-9dd7-a33f06a96422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bvjwt\" (UID: \"11b18d64-a832-43f3-9dd7-a33f06a96422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" Oct 29 23:40:08.037989 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.037973 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b18d64-a832-43f3-9dd7-a33f06a96422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bvjwt\" (UID: \"11b18d64-a832-43f3-9dd7-a33f06a96422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" Oct 29 23:40:08.047086 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.047065 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:40:08.056487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.056427 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:40:08.056487 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.056447 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:40:08.056685 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.056497 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:40:08.058586 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.058562 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt"] Oct 29 23:40:08.138615 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.138579 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b18d64-a832-43f3-9dd7-a33f06a96422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bvjwt\" (UID: \"11b18d64-a832-43f3-9dd7-a33f06a96422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" Oct 29 23:40:08.138615 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.138618 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b18d64-a832-43f3-9dd7-a33f06a96422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bvjwt\" (UID: \"11b18d64-a832-43f3-9dd7-a33f06a96422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" Oct 29 23:40:08.138838 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.138657 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvwsg\" (UniqueName: \"kubernetes.io/projected/11b18d64-a832-43f3-9dd7-a33f06a96422-kube-api-access-lvwsg\") pod \"maas-api-revoke-sub-d576e68b-bvjwt\" (UID: \"11b18d64-a832-43f3-9dd7-a33f06a96422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" Oct 29 23:40:08.139336 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.139317 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b18d64-a832-43f3-9dd7-a33f06a96422-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bvjwt\" (UID: \"11b18d64-a832-43f3-9dd7-a33f06a96422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" Oct 29 23:40:08.141036 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.141013 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b18d64-a832-43f3-9dd7-a33f06a96422-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bvjwt\" (UID: \"11b18d64-a832-43f3-9dd7-a33f06a96422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" Oct 29 23:40:08.155031 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.154995 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvwsg\" (UniqueName: \"kubernetes.io/projected/11b18d64-a832-43f3-9dd7-a33f06a96422-kube-api-access-lvwsg\") pod \"maas-api-revoke-sub-d576e68b-bvjwt\" (UID: \"11b18d64-a832-43f3-9dd7-a33f06a96422\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" Oct 29 23:40:08.347718 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.347634 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" Oct 29 23:40:08.638737 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:08.638625 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt"] Oct 29 23:40:08.643272 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:40:08.643237 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b18d64_a832_43f3_9dd7_a33f06a96422.slice/crio-2464faff5be27d39f17f16b47fc5623c2bbfaec76fd545e645a443991fbdd1a1 WatchSource:0}: Error finding container 2464faff5be27d39f17f16b47fc5623c2bbfaec76fd545e645a443991fbdd1a1: Status 404 returned error can't find the container with id 2464faff5be27d39f17f16b47fc5623c2bbfaec76fd545e645a443991fbdd1a1 Oct 29 23:40:09.386638 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:09.386591 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" event={"ID":"11b18d64-a832-43f3-9dd7-a33f06a96422","Type":"ContainerStarted","Data":"c853d88558d758e2a351989567405cd7534af14d0e01f9835f054342e0edca60"} Oct 29 23:40:09.387084 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:09.386659 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" event={"ID":"11b18d64-a832-43f3-9dd7-a33f06a96422","Type":"ContainerStarted","Data":"2464faff5be27d39f17f16b47fc5623c2bbfaec76fd545e645a443991fbdd1a1"} Oct 29 23:40:09.420563 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:09.420509 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" podStartSLOduration=1.420495719 podStartE2EDuration="1.420495719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:40:09.418566699 +0000 UTC m=+1578.781571422" watchObservedRunningTime="2026-10-29 23:40:09.420495719 +0000 UTC m=+1578.783500438" Oct 29 23:40:10.393733 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:10.393695 6507 generic.go:356] "Generic (PLEG): container finished" podID="11b18d64-a832-43f3-9dd7-a33f06a96422" containerID="c853d88558d758e2a351989567405cd7534af14d0e01f9835f054342e0edca60" exitCode=0 Oct 29 23:40:10.394109 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:10.393779 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" event={"ID":"11b18d64-a832-43f3-9dd7-a33f06a96422","Type":"ContainerDied","Data":"c853d88558d758e2a351989567405cd7534af14d0e01f9835f054342e0edca60"} Oct 29 23:40:11.542225 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:11.542198 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" Oct 29 23:40:11.566506 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:11.566449 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b18d64-a832-43f3-9dd7-a33f06a96422-kube-api-access-lvwsg\" (UniqueName: \"kubernetes.io/projected/11b18d64-a832-43f3-9dd7-a33f06a96422-kube-api-access-lvwsg\") pod \"11b18d64-a832-43f3-9dd7-a33f06a96422\" (UID: \"11b18d64-a832-43f3-9dd7-a33f06a96422\") " Oct 29 23:40:11.566650 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:11.566531 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11b18d64-a832-43f3-9dd7-a33f06a96422-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b18d64-a832-43f3-9dd7-a33f06a96422-maas-api-xks-ca\") pod \"11b18d64-a832-43f3-9dd7-a33f06a96422\" (UID: \"11b18d64-a832-43f3-9dd7-a33f06a96422\") " Oct 29 23:40:11.566650 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:11.566594 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b18d64-a832-43f3-9dd7-a33f06a96422-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b18d64-a832-43f3-9dd7-a33f06a96422-maas-api-openshift-ca\") pod \"11b18d64-a832-43f3-9dd7-a33f06a96422\" (UID: \"11b18d64-a832-43f3-9dd7-a33f06a96422\") " Oct 29 23:40:11.566988 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:11.566960 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b18d64-a832-43f3-9dd7-a33f06a96422-maas-api-openshift-ca" pod "11b18d64-a832-43f3-9dd7-a33f06a96422" (UID: "11b18d64-a832-43f3-9dd7-a33f06a96422"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:40:11.568690 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:11.568666 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11b18d64-a832-43f3-9dd7-a33f06a96422-maas-api-xks-ca" pod "11b18d64-a832-43f3-9dd7-a33f06a96422" (UID: "11b18d64-a832-43f3-9dd7-a33f06a96422"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:40:11.568690 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:11.568672 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b18d64-a832-43f3-9dd7-a33f06a96422-kube-api-access-lvwsg" pod "11b18d64-a832-43f3-9dd7-a33f06a96422" (UID: "11b18d64-a832-43f3-9dd7-a33f06a96422"). InnerVolumeSpecName "kube-api-access-lvwsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:40:11.668366 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:11.668272 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvwsg\" (UniqueName: \"kubernetes.io/projected/11b18d64-a832-43f3-9dd7-a33f06a96422-kube-api-access-lvwsg\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:40:11.668366 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:11.668309 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b18d64-a832-43f3-9dd7-a33f06a96422-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:40:11.668366 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:11.668319 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b18d64-a832-43f3-9dd7-a33f06a96422-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:40:12.406098 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:12.406066 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" Oct 29 23:40:12.406280 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:12.406101 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt" event={"ID":"11b18d64-a832-43f3-9dd7-a33f06a96422","Type":"ContainerDied","Data":"2464faff5be27d39f17f16b47fc5623c2bbfaec76fd545e645a443991fbdd1a1"} Oct 29 23:40:12.406280 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:12.406148 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2464faff5be27d39f17f16b47fc5623c2bbfaec76fd545e645a443991fbdd1a1" Oct 29 23:40:15.223659 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:40:15.223618 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:40:19.268341 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:19.268294 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:40:24.030483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.028663 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv"] Oct 29 23:40:24.039545 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.039500 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-8n6nv"] Oct 29 23:40:24.751475 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.751429 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee900d90-3aab-42a0-a9fd-57602324aae6" path="/var/lib/kubelet/pods/ee900d90-3aab-42a0-a9fd-57602324aae6/volumes" Oct 29 23:40:24.888036 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.888001 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz"] Oct 29 23:40:24.888510 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.888487 6507 cpu_manager.go:414] "containerMap: removing container" podUID="8c432c0b-430a-4beb-a88e-dbf7da35b599" containerName="istio-proxy" Oct 29 23:40:24.888686 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.888664 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c432c0b-430a-4beb-a88e-dbf7da35b599" containerName="istio-proxy" Oct 29 23:40:24.888686 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.888689 6507 cpu_manager.go:414] "containerMap: removing container" podUID="11b18d64-a832-43f3-9dd7-a33f06a96422" containerName="revoke-keys" Oct 29 23:40:24.888848 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.888698 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b18d64-a832-43f3-9dd7-a33f06a96422" containerName="revoke-keys" Oct 29 23:40:24.888898 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.888850 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b18d64-a832-43f3-9dd7-a33f06a96422" containerName="revoke-keys" Oct 29 23:40:24.892303 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.892286 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz" Oct 29 23:40:24.904451 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.904428 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:40:24.904695 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.904678 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:40:24.904752 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.904741 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:40:24.904817 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.904800 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:40:24.915865 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.915833 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz"] Oct 29 23:40:24.935219 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.935181 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7fts\" (UniqueName: \"kubernetes.io/projected/9d1de21b-4915-45ea-aa3b-552f45d473ef-kube-api-access-g7fts\") pod \"maas-api-revoke-sub-d2bc93bc-5w5rz\" (UID: \"9d1de21b-4915-45ea-aa3b-552f45d473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz" Oct 29 23:40:24.935404 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.935245 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1de21b-4915-45ea-aa3b-552f45d473ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5w5rz\" (UID: \"9d1de21b-4915-45ea-aa3b-552f45d473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz" Oct 29 23:40:24.935404 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:24.935279 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1de21b-4915-45ea-aa3b-552f45d473ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5w5rz\" (UID: \"9d1de21b-4915-45ea-aa3b-552f45d473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz" Oct 29 23:40:25.035858 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:25.035765 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7fts\" (UniqueName: \"kubernetes.io/projected/9d1de21b-4915-45ea-aa3b-552f45d473ef-kube-api-access-g7fts\") pod \"maas-api-revoke-sub-d2bc93bc-5w5rz\" (UID: \"9d1de21b-4915-45ea-aa3b-552f45d473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz" Oct 29 23:40:25.035858 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:25.035828 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1de21b-4915-45ea-aa3b-552f45d473ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5w5rz\" (UID: \"9d1de21b-4915-45ea-aa3b-552f45d473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz" Oct 29 23:40:25.036411 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:25.035887 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1de21b-4915-45ea-aa3b-552f45d473ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5w5rz\" (UID: \"9d1de21b-4915-45ea-aa3b-552f45d473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz" Oct 29 23:40:25.036686 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:25.036660 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1de21b-4915-45ea-aa3b-552f45d473ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5w5rz\" (UID: \"9d1de21b-4915-45ea-aa3b-552f45d473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz" Oct 29 23:40:25.038241 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:25.038222 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1de21b-4915-45ea-aa3b-552f45d473ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5w5rz\" (UID: \"9d1de21b-4915-45ea-aa3b-552f45d473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz" Oct 29 23:40:25.052514 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:25.052452 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7fts\" (UniqueName: \"kubernetes.io/projected/9d1de21b-4915-45ea-aa3b-552f45d473ef-kube-api-access-g7fts\") pod \"maas-api-revoke-sub-d2bc93bc-5w5rz\" (UID: \"9d1de21b-4915-45ea-aa3b-552f45d473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz" Oct 29 23:40:25.204657 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:25.204618 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz" Oct 29 23:40:25.500760 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:25.500726 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz"] Oct 29 23:40:25.504979 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:40:25.504953 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1de21b_4915_45ea_aa3b_552f45d473ef.slice/crio-e7cfb7059462c8cbc06f28a764d8257aa10602e06df4fec79ac762b6d8567b95 WatchSource:0}: Error finding container e7cfb7059462c8cbc06f28a764d8257aa10602e06df4fec79ac762b6d8567b95: Status 404 returned error can't find the container with id e7cfb7059462c8cbc06f28a764d8257aa10602e06df4fec79ac762b6d8567b95 Oct 29 23:40:26.500327 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:26.500290 6507 generic.go:356] "Generic (PLEG): container finished" podID="9d1de21b-4915-45ea-aa3b-552f45d473ef" containerID="0c0b4875d44e351e73f751685ec921068b29fa4935c65222470ff7dc83663210" exitCode=0 Oct 29 23:40:26.500728 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:26.500352 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz" event={"ID":"9d1de21b-4915-45ea-aa3b-552f45d473ef","Type":"ContainerDied","Data":"0c0b4875d44e351e73f751685ec921068b29fa4935c65222470ff7dc83663210"} Oct 29 23:40:26.500728 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:26.500392 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz" event={"ID":"9d1de21b-4915-45ea-aa3b-552f45d473ef","Type":"ContainerStarted","Data":"e7cfb7059462c8cbc06f28a764d8257aa10602e06df4fec79ac762b6d8567b95"} Oct 29 23:40:27.640090 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:27.640066 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz" Oct 29 23:40:27.663020 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:27.662995 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d1de21b-4915-45ea-aa3b-552f45d473ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1de21b-4915-45ea-aa3b-552f45d473ef-maas-api-xks-ca\") pod \"9d1de21b-4915-45ea-aa3b-552f45d473ef\" (UID: \"9d1de21b-4915-45ea-aa3b-552f45d473ef\") " Oct 29 23:40:27.663164 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:27.663135 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d1de21b-4915-45ea-aa3b-552f45d473ef-kube-api-access-g7fts\" (UniqueName: \"kubernetes.io/projected/9d1de21b-4915-45ea-aa3b-552f45d473ef-kube-api-access-g7fts\") pod \"9d1de21b-4915-45ea-aa3b-552f45d473ef\" (UID: \"9d1de21b-4915-45ea-aa3b-552f45d473ef\") " Oct 29 23:40:27.663217 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:27.663201 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d1de21b-4915-45ea-aa3b-552f45d473ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1de21b-4915-45ea-aa3b-552f45d473ef-maas-api-openshift-ca\") pod \"9d1de21b-4915-45ea-aa3b-552f45d473ef\" (UID: \"9d1de21b-4915-45ea-aa3b-552f45d473ef\") " Oct 29 23:40:27.663603 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:27.663574 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1de21b-4915-45ea-aa3b-552f45d473ef-maas-api-openshift-ca" pod "9d1de21b-4915-45ea-aa3b-552f45d473ef" (UID: "9d1de21b-4915-45ea-aa3b-552f45d473ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:40:27.665087 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:27.665066 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d1de21b-4915-45ea-aa3b-552f45d473ef-maas-api-xks-ca" pod "9d1de21b-4915-45ea-aa3b-552f45d473ef" (UID: "9d1de21b-4915-45ea-aa3b-552f45d473ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:40:27.665200 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:27.665180 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1de21b-4915-45ea-aa3b-552f45d473ef-kube-api-access-g7fts" pod "9d1de21b-4915-45ea-aa3b-552f45d473ef" (UID: "9d1de21b-4915-45ea-aa3b-552f45d473ef"). InnerVolumeSpecName "kube-api-access-g7fts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:40:27.764539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:27.764436 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d1de21b-4915-45ea-aa3b-552f45d473ef-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:40:27.764539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:27.764492 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7fts\" (UniqueName: \"kubernetes.io/projected/9d1de21b-4915-45ea-aa3b-552f45d473ef-kube-api-access-g7fts\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:40:27.764539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:27.764503 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d1de21b-4915-45ea-aa3b-552f45d473ef-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:40:28.511842 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:28.511810 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz" event={"ID":"9d1de21b-4915-45ea-aa3b-552f45d473ef","Type":"ContainerDied","Data":"e7cfb7059462c8cbc06f28a764d8257aa10602e06df4fec79ac762b6d8567b95"} Oct 29 23:40:28.511842 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:28.511838 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz" Oct 29 23:40:28.512054 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:28.511858 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7cfb7059462c8cbc06f28a764d8257aa10602e06df4fec79ac762b6d8567b95" Oct 29 23:40:37.031940 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.031907 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm"] Oct 29 23:40:37.032522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.032308 6507 cpu_manager.go:414] "containerMap: removing container" podUID="9d1de21b-4915-45ea-aa3b-552f45d473ef" containerName="revoke-keys" Oct 29 23:40:37.032522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.032319 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d1de21b-4915-45ea-aa3b-552f45d473ef" containerName="revoke-keys" Oct 29 23:40:37.032522 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.032441 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d1de21b-4915-45ea-aa3b-552f45d473ef" containerName="revoke-keys" Oct 29 23:40:37.036066 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.036047 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.050199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.050175 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-e165bd-openshift-default-dockercfg-5wjrz\"" Oct 29 23:40:37.062745 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.062715 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm"] Oct 29 23:40:37.152932 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.152824 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741be03f-82a0-4001-a343-84f8f8a028f5-istio-token\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.153240 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.153215 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-istio-data\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.153453 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.153431 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741be03f-82a0-4001-a343-84f8f8a028f5-istio-podinfo\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.153733 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.153713 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-credential-socket\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.153887 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.153861 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plt8n\" (UniqueName: \"kubernetes.io/projected/741be03f-82a0-4001-a343-84f8f8a028f5-kube-api-access-plt8n\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.154013 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.153898 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741be03f-82a0-4001-a343-84f8f8a028f5-istiod-ca-cert\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.154013 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.153935 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-workload-certs\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.154150 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.154027 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-istio-envoy\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.154150 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.154076 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-workload-socket\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.255374 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.255341 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741be03f-82a0-4001-a343-84f8f8a028f5-istio-podinfo\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.255621 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.255390 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-credential-socket\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.255621 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.255486 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plt8n\" (UniqueName: \"kubernetes.io/projected/741be03f-82a0-4001-a343-84f8f8a028f5-kube-api-access-plt8n\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.255621 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.255506 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741be03f-82a0-4001-a343-84f8f8a028f5-istiod-ca-cert\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.255621 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.255525 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-workload-certs\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.255621 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.255545 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-istio-envoy\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.255621 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.255575 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-workload-socket\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.255969 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.255641 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741be03f-82a0-4001-a343-84f8f8a028f5-istio-token\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.255969 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.255685 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-istio-data\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.256069 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.256014 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-workload-certs\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.256125 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.256068 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-istio-data\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.256186 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.256134 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-workload-socket\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.256312 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.256293 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-credential-socket\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.256374 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.256310 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741be03f-82a0-4001-a343-84f8f8a028f5-istiod-ca-cert\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.258065 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.258043 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-istio-envoy\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.258065 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.258055 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741be03f-82a0-4001-a343-84f8f8a028f5-istio-podinfo\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.272849 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.272811 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741be03f-82a0-4001-a343-84f8f8a028f5-istio-token\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.273287 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.273263 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plt8n\" (UniqueName: \"kubernetes.io/projected/741be03f-82a0-4001-a343-84f8f8a028f5-kube-api-access-plt8n\") pod \"e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.353062 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.352974 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:37.663556 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.663472 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm"] Oct 29 23:40:37.667046 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:40:37.667009 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741be03f_82a0_4001_a343_84f8f8a028f5.slice/crio-b38da56d83d4173150d26739db241206e7aea78211e61bd4f4236de9de11fdf5 WatchSource:0}: Error finding container b38da56d83d4173150d26739db241206e7aea78211e61bd4f4236de9de11fdf5: Status 404 returned error can't find the container with id b38da56d83d4173150d26739db241206e7aea78211e61bd4f4236de9de11fdf5 Oct 29 23:40:37.669246 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.669213 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:40:37.669318 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.669281 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:40:37.669318 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:37.669309 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:40:38.563333 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:38.563294 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" event={"ID":"741be03f-82a0-4001-a343-84f8f8a028f5","Type":"ContainerStarted","Data":"cdfa415defc0087512005144d2f60fd326ed0e32c6cfa840c7a47c091d23c2a3"} Oct 29 23:40:38.563868 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:38.563354 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" event={"ID":"741be03f-82a0-4001-a343-84f8f8a028f5","Type":"ContainerStarted","Data":"b38da56d83d4173150d26739db241206e7aea78211e61bd4f4236de9de11fdf5"} Oct 29 23:40:38.610851 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:38.610787 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" podStartSLOduration=1.610772745 podStartE2EDuration="1.610772745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:40:38.609243314 +0000 UTC m=+1607.972248037" watchObservedRunningTime="2026-10-29 23:40:38.610772745 +0000 UTC m=+1607.973777535" Oct 29 23:40:39.354201 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:39.354166 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:39.359153 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:39.359126 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:39.568202 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:39.568171 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:39.569147 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:39.569127 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:42.966056 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:42.966016 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k"] Oct 29 23:40:42.969827 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:42.969806 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" Oct 29 23:40:42.981914 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:42.981888 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-e165bd\"" Oct 29 23:40:42.993695 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:42.993657 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k"] Oct 29 23:40:43.030031 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.029971 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9b88a-d944-44bc-b044-d1c80fca0555-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k\" (UID: \"33b9b88a-d944-44bc-b044-d1c80fca0555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" Oct 29 23:40:43.030233 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.030040 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssrnv\" (UniqueName: \"kubernetes.io/projected/33b9b88a-d944-44bc-b044-d1c80fca0555-kube-api-access-ssrnv\") pod \"payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k\" (UID: \"33b9b88a-d944-44bc-b044-d1c80fca0555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" Oct 29 23:40:43.037509 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.037450 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-531d1fc0-dockercfg-kgh8q\"" Oct 29 23:40:43.052846 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.052775 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6"] Oct 29 23:40:43.057798 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.057772 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" Oct 29 23:40:43.087729 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.087626 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6"] Oct 29 23:40:43.131534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.131497 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880367dc-1403-444f-8b2c-cf9d6cf2dbf1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6\" (UID: \"880367dc-1403-444f-8b2c-cf9d6cf2dbf1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" Oct 29 23:40:43.131719 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.131541 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9b88a-d944-44bc-b044-d1c80fca0555-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k\" (UID: \"33b9b88a-d944-44bc-b044-d1c80fca0555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" Oct 29 23:40:43.131719 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.131628 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssrnv\" (UniqueName: \"kubernetes.io/projected/33b9b88a-d944-44bc-b044-d1c80fca0555-kube-api-access-ssrnv\") pod \"payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k\" (UID: \"33b9b88a-d944-44bc-b044-d1c80fca0555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" Oct 29 23:40:43.131719 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.131706 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75558\" (UniqueName: \"kubernetes.io/projected/880367dc-1403-444f-8b2c-cf9d6cf2dbf1-kube-api-access-75558\") pod \"payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6\" (UID: \"880367dc-1403-444f-8b2c-cf9d6cf2dbf1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" Oct 29 23:40:43.132301 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.132283 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9b88a-d944-44bc-b044-d1c80fca0555-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k\" (UID: \"33b9b88a-d944-44bc-b044-d1c80fca0555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" Oct 29 23:40:43.148255 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.148218 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssrnv\" (UniqueName: \"kubernetes.io/projected/33b9b88a-d944-44bc-b044-d1c80fca0555-kube-api-access-ssrnv\") pod \"payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k\" (UID: \"33b9b88a-d944-44bc-b044-d1c80fca0555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" Oct 29 23:40:43.232986 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.232901 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880367dc-1403-444f-8b2c-cf9d6cf2dbf1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6\" (UID: \"880367dc-1403-444f-8b2c-cf9d6cf2dbf1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" Oct 29 23:40:43.232986 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.232959 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75558\" (UniqueName: \"kubernetes.io/projected/880367dc-1403-444f-8b2c-cf9d6cf2dbf1-kube-api-access-75558\") pod \"payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6\" (UID: \"880367dc-1403-444f-8b2c-cf9d6cf2dbf1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" Oct 29 23:40:43.233944 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.233918 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880367dc-1403-444f-8b2c-cf9d6cf2dbf1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6\" (UID: \"880367dc-1403-444f-8b2c-cf9d6cf2dbf1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" Oct 29 23:40:43.249361 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.249330 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75558\" (UniqueName: \"kubernetes.io/projected/880367dc-1403-444f-8b2c-cf9d6cf2dbf1-kube-api-access-75558\") pod \"payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6\" (UID: \"880367dc-1403-444f-8b2c-cf9d6cf2dbf1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" Oct 29 23:40:43.286275 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.286246 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" Oct 29 23:40:43.397514 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.396780 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" Oct 29 23:40:43.579882 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.579796 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k"] Oct 29 23:40:43.583573 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:40:43.583545 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b9b88a_d944_44bc_b044_d1c80fca0555.slice/crio-cf83beb120743bb6fe67d6b9e47bb9b4309dc292994b8f10bf5a80225be9d062 WatchSource:0}: Error finding container cf83beb120743bb6fe67d6b9e47bb9b4309dc292994b8f10bf5a80225be9d062: Status 404 returned error can't find the container with id cf83beb120743bb6fe67d6b9e47bb9b4309dc292994b8f10bf5a80225be9d062 Oct 29 23:40:43.594789 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.594754 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" event={"ID":"33b9b88a-d944-44bc-b044-d1c80fca0555","Type":"ContainerStarted","Data":"cf83beb120743bb6fe67d6b9e47bb9b4309dc292994b8f10bf5a80225be9d062"} Oct 29 23:40:43.703203 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:43.699927 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6"] Oct 29 23:40:43.709384 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:40:43.709356 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880367dc_1403_444f_8b2c_cf9d6cf2dbf1.slice/crio-5dd8d909e629b3710908ad4f920de7ca436b2a4597079c001816954d47db6c8e WatchSource:0}: Error finding container 5dd8d909e629b3710908ad4f920de7ca436b2a4597079c001816954d47db6c8e: Status 404 returned error can't find the container with id 5dd8d909e629b3710908ad4f920de7ca436b2a4597079c001816954d47db6c8e Oct 29 23:40:44.614301 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:44.614262 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" event={"ID":"880367dc-1403-444f-8b2c-cf9d6cf2dbf1","Type":"ContainerStarted","Data":"aec0e658f1f4b9d243682d0a5af3e0c6a200209936b1ec4824ece95f1f476ad9"} Oct 29 23:40:44.614808 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:44.614336 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" event={"ID":"880367dc-1403-444f-8b2c-cf9d6cf2dbf1","Type":"ContainerStarted","Data":"5dd8d909e629b3710908ad4f920de7ca436b2a4597079c001816954d47db6c8e"} Oct 29 23:40:44.614808 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:44.614369 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" Oct 29 23:40:44.615893 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:44.615867 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" event={"ID":"33b9b88a-d944-44bc-b044-d1c80fca0555","Type":"ContainerStarted","Data":"56bf39a3a3cb019e0efb92d3c2865fb8ba231057f5ae67c97293f1609b2928c3"} Oct 29 23:40:44.616027 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:44.615927 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" Oct 29 23:40:44.649408 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:44.649340 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" podStartSLOduration=1.6493278120000001 podStartE2EDuration="1.649327812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:40:44.647488715 +0000 UTC m=+1614.010493432" watchObservedRunningTime="2026-10-29 23:40:44.649327812 +0000 UTC m=+1614.012332532" Oct 29 23:40:44.680817 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:44.680748 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" podStartSLOduration=2.680732829 podStartE2EDuration="2.680732829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:40:44.679900473 +0000 UTC m=+1614.042905196" watchObservedRunningTime="2026-10-29 23:40:44.680732829 +0000 UTC m=+1614.043737554" Oct 29 23:40:45.024920 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:45.024879 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz"] Oct 29 23:40:45.027884 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:45.027858 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rptjz"] Oct 29 23:40:45.223143 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:40:45.223101 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:40:46.062847 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:46.062809 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:40:46.752344 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:46.752312 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a46803-d1ee-4689-8030-c2087e98f4ef" path="/var/lib/kubelet/pods/90a46803-d1ee-4689-8030-c2087e98f4ef/volumes" Oct 29 23:40:53.229718 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:53.229678 6507 scope.go:122] "RemoveContainer" containerID="22d6821168d052fd409abab6659a4db557dc4ff1c66e76bcb35ac51d4405c7f9" Oct 29 23:40:53.238689 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:53.238666 6507 scope.go:122] "RemoveContainer" containerID="a007326d8e904a97b20b826ff2c86460d9da9e2ccda1e7c75ac6cb990cd8fad6" Oct 29 23:40:53.247006 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:53.246981 6507 scope.go:122] "RemoveContainer" containerID="851d5ce1f272f113fcbb6307b3e514a4d998f80cdf4e91b39dc62e684f28ba85" Oct 29 23:40:55.471188 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:55.470394 6507 patch_prober.go:29] interesting pod/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": read tcp 10.133.0.2:46672->10.133.0.142:15021: read: connection reset by peer" start-of-body= Oct 29 23:40:55.471188 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:55.470504 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" podUID="741be03f-82a0-4001-a343-84f8f8a028f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": read tcp 10.133.0.2:46672->10.133.0.142:15021: read: connection reset by peer" Oct 29 23:40:55.623421 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:55.623345 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" Oct 29 23:40:55.623421 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:55.623408 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" Oct 29 23:40:55.688337 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:55.688303 6507 generic.go:356] "Generic (PLEG): container finished" podID="741be03f-82a0-4001-a343-84f8f8a028f5" containerID="cdfa415defc0087512005144d2f60fd326ed0e32c6cfa840c7a47c091d23c2a3" exitCode=0 Oct 29 23:40:55.688867 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:55.688384 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" event={"ID":"741be03f-82a0-4001-a343-84f8f8a028f5","Type":"ContainerDied","Data":"cdfa415defc0087512005144d2f60fd326ed0e32c6cfa840c7a47c091d23c2a3"} Oct 29 23:40:55.688867 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:55.688790 6507 scope.go:122] "RemoveContainer" containerID="cdfa415defc0087512005144d2f60fd326ed0e32c6cfa840c7a47c091d23c2a3" Oct 29 23:40:55.690307 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:55.690273 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:40:55.690419 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:55.690348 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:40:55.690419 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:55.690388 6507 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 29 23:40:56.696018 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:56.695980 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" event={"ID":"741be03f-82a0-4001-a343-84f8f8a028f5","Type":"ContainerStarted","Data":"f797b04c10c6dbb37d945ce8936fbfa97345ffea9ab3f5c54d2e05703c614b60"} Oct 29 23:40:57.353279 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:57.353238 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:40:57.353490 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:57.353436 6507 patch_prober.go:29] interesting pod/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:40:57.353543 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:57.353511 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" podUID="741be03f-82a0-4001-a343-84f8f8a028f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 29 23:40:58.090677 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.090639 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55"] Oct 29 23:40:58.095629 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.095594 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.142546 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.142510 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-e165bd\"/\"simulator-single-main-e165bd-kserve-self-signed-certs\"" Oct 29 23:40:58.142546 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.142539 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-e165bd\"/\"openshift-service-ca.crt\"" Oct 29 23:40:58.142806 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.142551 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-e165bd\"/\"default-dockercfg-76ddx\"" Oct 29 23:40:58.142806 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.142573 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-e165bd\"/\"kube-root-ca.crt\"" Oct 29 23:40:58.144650 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.144616 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55"] Oct 29 23:40:58.233675 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.233632 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-tmp-dir\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.233877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.233711 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-dshm\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.233877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.233740 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-home\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.233877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.233768 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfhz4\" (UniqueName: \"kubernetes.io/projected/9a8bb1d9-aad2-48a8-b589-13ee90356eff-kube-api-access-pfhz4\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.233985 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.233885 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8bb1d9-aad2-48a8-b589-13ee90356eff-tls-certs\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.233985 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.233924 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-model-cache\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.335256 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.335214 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8bb1d9-aad2-48a8-b589-13ee90356eff-tls-certs\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.335256 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.335258 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-model-cache\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.335526 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.335291 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-tmp-dir\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.335526 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.335344 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-dshm\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.335526 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.335377 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-home\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.335526 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.335416 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfhz4\" (UniqueName: \"kubernetes.io/projected/9a8bb1d9-aad2-48a8-b589-13ee90356eff-kube-api-access-pfhz4\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.335849 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.335819 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-model-cache\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.335951 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.335828 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-home\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.335951 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.335882 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-tmp-dir\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.337722 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.337697 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-dshm\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.338094 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.338074 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8bb1d9-aad2-48a8-b589-13ee90356eff-tls-certs\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.353267 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.353210 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfhz4\" (UniqueName: \"kubernetes.io/projected/9a8bb1d9-aad2-48a8-b589-13ee90356eff-kube-api-access-pfhz4\") pod \"simulator-single-main-e165bd-kserve-5464785447-zsp55\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.353591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.353570 6507 patch_prober.go:29] interesting pod/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:40:58.353687 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.353633 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" podUID="741be03f-82a0-4001-a343-84f8f8a028f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 29 23:40:58.440482 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.439620 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:40:58.779153 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:58.779109 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55"] Oct 29 23:40:58.783419 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:40:58.783378 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a8bb1d9_aad2_48a8_b589_13ee90356eff.slice/crio-0019f9322e5c9ded7d998c2baeda525cdc5acd6de44e511e4d4f8154811115a3 WatchSource:0}: Error finding container 0019f9322e5c9ded7d998c2baeda525cdc5acd6de44e511e4d4f8154811115a3: Status 404 returned error can't find the container with id 0019f9322e5c9ded7d998c2baeda525cdc5acd6de44e511e4d4f8154811115a3 Oct 29 23:40:59.353868 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:59.353833 6507 patch_prober.go:29] interesting pod/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:40:59.354258 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:59.353903 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" podUID="741be03f-82a0-4001-a343-84f8f8a028f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 29 23:40:59.740168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:59.740131 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" event={"ID":"9a8bb1d9-aad2-48a8-b589-13ee90356eff","Type":"ContainerStarted","Data":"4caef3f1fb765c40f172302167ed98589d2cd3f02e43ff891eccd9beefda8eee"} Oct 29 23:40:59.740343 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:59.740179 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" event={"ID":"9a8bb1d9-aad2-48a8-b589-13ee90356eff","Type":"ContainerStarted","Data":"0019f9322e5c9ded7d998c2baeda525cdc5acd6de44e511e4d4f8154811115a3"} Oct 29 23:40:59.778574 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:40:59.778522 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" podStartSLOduration=1.778507005 podStartE2EDuration="1.778507005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:40:59.776796318 +0000 UTC m=+1629.139801041" watchObservedRunningTime="2026-10-29 23:40:59.778507005 +0000 UTC m=+1629.141511719" Oct 29 23:41:00.353450 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:00.353403 6507 patch_prober.go:29] interesting pod/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:41:00.353645 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:00.353475 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" podUID="741be03f-82a0-4001-a343-84f8f8a028f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 29 23:41:00.417492 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:00.417437 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:41:00.440878 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:00.440843 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:41:01.353425 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:01.353379 6507 patch_prober.go:29] interesting pod/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:41:01.353646 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:01.353480 6507 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" podUID="741be03f-82a0-4001-a343-84f8f8a028f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 29 23:41:02.362142 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:02.362110 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:41:02.362630 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:02.362395 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:41:02.363130 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:02.363113 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:41:05.025336 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:05.025298 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw"] Oct 29 23:41:05.034583 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:05.034553 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-675nw"] Oct 29 23:41:06.751866 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:06.751831 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d88e96d-992e-4d89-9501-b9cbf8722f66" path="/var/lib/kubelet/pods/0d88e96d-992e-4d89-9501-b9cbf8722f66/volumes" Oct 29 23:41:08.440582 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:08.440551 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:41:08.453254 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:08.453226 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:41:08.804686 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:08.804598 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:41:15.226967 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:41:15.226926 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:41:15.601949 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:15.601863 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:41:17.743312 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:17.743276 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh"] Oct 29 23:41:17.747435 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:17.747405 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" Oct 29 23:41:17.773619 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:17.773595 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:41:17.773729 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:17.773616 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:41:17.773729 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:17.773617 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:41:17.773729 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:17.773623 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:41:17.776032 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:17.776011 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh"] Oct 29 23:41:17.838998 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:17.838962 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg6d4\" (UniqueName: \"kubernetes.io/projected/efab58b1-884e-4508-9ee9-3ca2d57d8680-kube-api-access-lg6d4\") pod \"maas-api-revoke-sub-cc36b4a1-jdsdh\" (UID: \"efab58b1-884e-4508-9ee9-3ca2d57d8680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" Oct 29 23:41:17.839318 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:17.839297 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efab58b1-884e-4508-9ee9-3ca2d57d8680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc36b4a1-jdsdh\" (UID: \"efab58b1-884e-4508-9ee9-3ca2d57d8680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" Oct 29 23:41:17.839649 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:17.839626 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efab58b1-884e-4508-9ee9-3ca2d57d8680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc36b4a1-jdsdh\" (UID: \"efab58b1-884e-4508-9ee9-3ca2d57d8680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" Oct 29 23:41:17.940907 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:17.940854 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg6d4\" (UniqueName: \"kubernetes.io/projected/efab58b1-884e-4508-9ee9-3ca2d57d8680-kube-api-access-lg6d4\") pod \"maas-api-revoke-sub-cc36b4a1-jdsdh\" (UID: \"efab58b1-884e-4508-9ee9-3ca2d57d8680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" Oct 29 23:41:17.941090 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:17.940943 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efab58b1-884e-4508-9ee9-3ca2d57d8680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc36b4a1-jdsdh\" (UID: \"efab58b1-884e-4508-9ee9-3ca2d57d8680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" Oct 29 23:41:17.941090 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:17.941013 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efab58b1-884e-4508-9ee9-3ca2d57d8680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc36b4a1-jdsdh\" (UID: \"efab58b1-884e-4508-9ee9-3ca2d57d8680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" Oct 29 23:41:17.941674 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:17.941655 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efab58b1-884e-4508-9ee9-3ca2d57d8680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc36b4a1-jdsdh\" (UID: \"efab58b1-884e-4508-9ee9-3ca2d57d8680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" Oct 29 23:41:17.943473 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:17.943434 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efab58b1-884e-4508-9ee9-3ca2d57d8680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc36b4a1-jdsdh\" (UID: \"efab58b1-884e-4508-9ee9-3ca2d57d8680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" Oct 29 23:41:17.958060 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:17.958034 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg6d4\" (UniqueName: \"kubernetes.io/projected/efab58b1-884e-4508-9ee9-3ca2d57d8680-kube-api-access-lg6d4\") pod \"maas-api-revoke-sub-cc36b4a1-jdsdh\" (UID: \"efab58b1-884e-4508-9ee9-3ca2d57d8680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" Oct 29 23:41:18.060020 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:18.059938 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" Oct 29 23:41:18.354223 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:18.354086 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh"] Oct 29 23:41:18.358359 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:41:18.358333 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefab58b1_884e_4508_9ee9_3ca2d57d8680.slice/crio-39c45d96cd03c0ce73e5f532ac2617b130b188e7cd5f35abf29c29b9284e359a WatchSource:0}: Error finding container 39c45d96cd03c0ce73e5f532ac2617b130b188e7cd5f35abf29c29b9284e359a: Status 404 returned error can't find the container with id 39c45d96cd03c0ce73e5f532ac2617b130b188e7cd5f35abf29c29b9284e359a Oct 29 23:41:18.848284 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:18.848250 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" event={"ID":"efab58b1-884e-4508-9ee9-3ca2d57d8680","Type":"ContainerStarted","Data":"d766985d08231719852f395e4d63892af887370377d2dd1028ec042cf215d34d"} Oct 29 23:41:18.848705 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:18.848299 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" event={"ID":"efab58b1-884e-4508-9ee9-3ca2d57d8680","Type":"ContainerStarted","Data":"39c45d96cd03c0ce73e5f532ac2617b130b188e7cd5f35abf29c29b9284e359a"} Oct 29 23:41:18.882243 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:18.882145 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" podStartSLOduration=1.8821322619999998 podStartE2EDuration="1.882132262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:41:18.880880202 +0000 UTC m=+1648.243884925" watchObservedRunningTime="2026-10-29 23:41:18.882132262 +0000 UTC m=+1648.245137009" Oct 29 23:41:19.854994 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:19.854963 6507 generic.go:356] "Generic (PLEG): container finished" podID="efab58b1-884e-4508-9ee9-3ca2d57d8680" containerID="d766985d08231719852f395e4d63892af887370377d2dd1028ec042cf215d34d" exitCode=0 Oct 29 23:41:19.855418 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:19.855003 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" event={"ID":"efab58b1-884e-4508-9ee9-3ca2d57d8680","Type":"ContainerDied","Data":"d766985d08231719852f395e4d63892af887370377d2dd1028ec042cf215d34d"} Oct 29 23:41:21.001566 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:21.001540 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" Oct 29 23:41:21.082298 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:21.082269 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efab58b1-884e-4508-9ee9-3ca2d57d8680-kube-api-access-lg6d4\" (UniqueName: \"kubernetes.io/projected/efab58b1-884e-4508-9ee9-3ca2d57d8680-kube-api-access-lg6d4\") pod \"efab58b1-884e-4508-9ee9-3ca2d57d8680\" (UID: \"efab58b1-884e-4508-9ee9-3ca2d57d8680\") " Oct 29 23:41:21.082486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:21.082306 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efab58b1-884e-4508-9ee9-3ca2d57d8680-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efab58b1-884e-4508-9ee9-3ca2d57d8680-maas-api-xks-ca\") pod \"efab58b1-884e-4508-9ee9-3ca2d57d8680\" (UID: \"efab58b1-884e-4508-9ee9-3ca2d57d8680\") " Oct 29 23:41:21.082486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:21.082388 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efab58b1-884e-4508-9ee9-3ca2d57d8680-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efab58b1-884e-4508-9ee9-3ca2d57d8680-maas-api-openshift-ca\") pod \"efab58b1-884e-4508-9ee9-3ca2d57d8680\" (UID: \"efab58b1-884e-4508-9ee9-3ca2d57d8680\") " Oct 29 23:41:21.082815 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:21.082783 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efab58b1-884e-4508-9ee9-3ca2d57d8680-maas-api-openshift-ca" pod "efab58b1-884e-4508-9ee9-3ca2d57d8680" (UID: "efab58b1-884e-4508-9ee9-3ca2d57d8680"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:41:21.084524 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:21.084497 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efab58b1-884e-4508-9ee9-3ca2d57d8680-kube-api-access-lg6d4" pod "efab58b1-884e-4508-9ee9-3ca2d57d8680" (UID: "efab58b1-884e-4508-9ee9-3ca2d57d8680"). InnerVolumeSpecName "kube-api-access-lg6d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:41:21.084624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:21.084585 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efab58b1-884e-4508-9ee9-3ca2d57d8680-maas-api-xks-ca" pod "efab58b1-884e-4508-9ee9-3ca2d57d8680" (UID: "efab58b1-884e-4508-9ee9-3ca2d57d8680"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:41:21.184316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:21.184223 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg6d4\" (UniqueName: \"kubernetes.io/projected/efab58b1-884e-4508-9ee9-3ca2d57d8680-kube-api-access-lg6d4\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:21.184316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:21.184265 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efab58b1-884e-4508-9ee9-3ca2d57d8680-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:21.184316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:21.184280 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efab58b1-884e-4508-9ee9-3ca2d57d8680-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:21.867940 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:21.867910 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" Oct 29 23:41:21.868126 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:21.867903 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh" event={"ID":"efab58b1-884e-4508-9ee9-3ca2d57d8680","Type":"ContainerDied","Data":"39c45d96cd03c0ce73e5f532ac2617b130b188e7cd5f35abf29c29b9284e359a"} Oct 29 23:41:21.868126 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:21.868040 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39c45d96cd03c0ce73e5f532ac2617b130b188e7cd5f35abf29c29b9284e359a" Oct 29 23:41:28.759486 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.759432 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx"] Oct 29 23:41:28.759968 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.759947 6507 cpu_manager.go:414] "containerMap: removing container" podUID="efab58b1-884e-4508-9ee9-3ca2d57d8680" containerName="revoke-keys" Oct 29 23:41:28.760039 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.759971 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efab58b1-884e-4508-9ee9-3ca2d57d8680" containerName="revoke-keys" Oct 29 23:41:28.760116 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.760104 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="efab58b1-884e-4508-9ee9-3ca2d57d8680" containerName="revoke-keys" Oct 29 23:41:28.763858 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.763840 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" Oct 29 23:41:28.785722 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.785694 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:41:28.785849 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.785830 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:41:28.785911 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.785877 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:41:28.785911 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.785887 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:41:28.796034 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.796005 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx"] Oct 29 23:41:28.862401 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.862365 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfj6l\" (UniqueName: \"kubernetes.io/projected/66168d46-aaea-4034-8236-4e9d6a35c4a3-kube-api-access-pfj6l\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx\" (UID: \"66168d46-aaea-4034-8236-4e9d6a35c4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" Oct 29 23:41:28.862587 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.862415 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66168d46-aaea-4034-8236-4e9d6a35c4a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx\" (UID: \"66168d46-aaea-4034-8236-4e9d6a35c4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" Oct 29 23:41:28.862642 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.862606 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66168d46-aaea-4034-8236-4e9d6a35c4a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx\" (UID: \"66168d46-aaea-4034-8236-4e9d6a35c4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" Oct 29 23:41:28.963835 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.963799 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66168d46-aaea-4034-8236-4e9d6a35c4a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx\" (UID: \"66168d46-aaea-4034-8236-4e9d6a35c4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" Oct 29 23:41:28.964031 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.963939 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfj6l\" (UniqueName: \"kubernetes.io/projected/66168d46-aaea-4034-8236-4e9d6a35c4a3-kube-api-access-pfj6l\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx\" (UID: \"66168d46-aaea-4034-8236-4e9d6a35c4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" Oct 29 23:41:28.964031 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.963975 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66168d46-aaea-4034-8236-4e9d6a35c4a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx\" (UID: \"66168d46-aaea-4034-8236-4e9d6a35c4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" Oct 29 23:41:28.964568 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.964543 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66168d46-aaea-4034-8236-4e9d6a35c4a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx\" (UID: \"66168d46-aaea-4034-8236-4e9d6a35c4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" Oct 29 23:41:28.966452 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.966430 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66168d46-aaea-4034-8236-4e9d6a35c4a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx\" (UID: \"66168d46-aaea-4034-8236-4e9d6a35c4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" Oct 29 23:41:28.981125 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:28.981095 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfj6l\" (UniqueName: \"kubernetes.io/projected/66168d46-aaea-4034-8236-4e9d6a35c4a3-kube-api-access-pfj6l\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx\" (UID: \"66168d46-aaea-4034-8236-4e9d6a35c4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" Oct 29 23:41:29.071818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:29.071719 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:41:29.079313 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:29.079286 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" Oct 29 23:41:29.376285 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:29.376192 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx"] Oct 29 23:41:29.396791 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:41:29.396756 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66168d46_aaea_4034_8236_4e9d6a35c4a3.slice/crio-17c4d10b05559ad1fe359d983a947df49ffe8ab43afe8bf3f6c6b0236d43bc68 WatchSource:0}: Error finding container 17c4d10b05559ad1fe359d983a947df49ffe8ab43afe8bf3f6c6b0236d43bc68: Status 404 returned error can't find the container with id 17c4d10b05559ad1fe359d983a947df49ffe8ab43afe8bf3f6c6b0236d43bc68 Oct 29 23:41:29.912733 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:29.912689 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" event={"ID":"66168d46-aaea-4034-8236-4e9d6a35c4a3","Type":"ContainerStarted","Data":"125828c7085ef22e9438f3af9cf6af5cdbb7c8a00a1564b6f17c78026a9ee62b"} Oct 29 23:41:29.913260 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:29.912756 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" event={"ID":"66168d46-aaea-4034-8236-4e9d6a35c4a3","Type":"ContainerStarted","Data":"17c4d10b05559ad1fe359d983a947df49ffe8ab43afe8bf3f6c6b0236d43bc68"} Oct 29 23:41:29.944706 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:29.944655 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" podStartSLOduration=1.944640347 podStartE2EDuration="1.944640347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:41:29.94455768 +0000 UTC m=+1659.307562404" watchObservedRunningTime="2026-10-29 23:41:29.944640347 +0000 UTC m=+1659.307645070" Oct 29 23:41:30.920048 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:30.920010 6507 generic.go:356] "Generic (PLEG): container finished" podID="66168d46-aaea-4034-8236-4e9d6a35c4a3" containerID="125828c7085ef22e9438f3af9cf6af5cdbb7c8a00a1564b6f17c78026a9ee62b" exitCode=0 Oct 29 23:41:30.920441 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:30.920052 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" event={"ID":"66168d46-aaea-4034-8236-4e9d6a35c4a3","Type":"ContainerDied","Data":"125828c7085ef22e9438f3af9cf6af5cdbb7c8a00a1564b6f17c78026a9ee62b"} Oct 29 23:41:32.085291 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:32.085255 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" Oct 29 23:41:32.095619 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:32.095591 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66168d46-aaea-4034-8236-4e9d6a35c4a3-kube-api-access-pfj6l\" (UniqueName: \"kubernetes.io/projected/66168d46-aaea-4034-8236-4e9d6a35c4a3-kube-api-access-pfj6l\") pod \"66168d46-aaea-4034-8236-4e9d6a35c4a3\" (UID: \"66168d46-aaea-4034-8236-4e9d6a35c4a3\") " Oct 29 23:41:32.095786 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:32.095627 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66168d46-aaea-4034-8236-4e9d6a35c4a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66168d46-aaea-4034-8236-4e9d6a35c4a3-maas-api-openshift-ca\") pod \"66168d46-aaea-4034-8236-4e9d6a35c4a3\" (UID: \"66168d46-aaea-4034-8236-4e9d6a35c4a3\") " Oct 29 23:41:32.095786 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:32.095679 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66168d46-aaea-4034-8236-4e9d6a35c4a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66168d46-aaea-4034-8236-4e9d6a35c4a3-maas-api-xks-ca\") pod \"66168d46-aaea-4034-8236-4e9d6a35c4a3\" (UID: \"66168d46-aaea-4034-8236-4e9d6a35c4a3\") " Oct 29 23:41:32.096027 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:32.096004 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66168d46-aaea-4034-8236-4e9d6a35c4a3-maas-api-openshift-ca" pod "66168d46-aaea-4034-8236-4e9d6a35c4a3" (UID: "66168d46-aaea-4034-8236-4e9d6a35c4a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:41:32.097781 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:32.097756 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66168d46-aaea-4034-8236-4e9d6a35c4a3-maas-api-xks-ca" pod "66168d46-aaea-4034-8236-4e9d6a35c4a3" (UID: "66168d46-aaea-4034-8236-4e9d6a35c4a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:41:32.098025 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:32.098007 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66168d46-aaea-4034-8236-4e9d6a35c4a3-kube-api-access-pfj6l" pod "66168d46-aaea-4034-8236-4e9d6a35c4a3" (UID: "66168d46-aaea-4034-8236-4e9d6a35c4a3"). InnerVolumeSpecName "kube-api-access-pfj6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:41:32.196789 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:32.196759 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfj6l\" (UniqueName: \"kubernetes.io/projected/66168d46-aaea-4034-8236-4e9d6a35c4a3-kube-api-access-pfj6l\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:32.196789 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:32.196788 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66168d46-aaea-4034-8236-4e9d6a35c4a3-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:32.196982 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:32.196798 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66168d46-aaea-4034-8236-4e9d6a35c4a3-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:32.930795 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:32.930765 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" Oct 29 23:41:32.930976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:32.930798 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx" event={"ID":"66168d46-aaea-4034-8236-4e9d6a35c4a3","Type":"ContainerDied","Data":"17c4d10b05559ad1fe359d983a947df49ffe8ab43afe8bf3f6c6b0236d43bc68"} Oct 29 23:41:32.930976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:32.930839 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17c4d10b05559ad1fe359d983a947df49ffe8ab43afe8bf3f6c6b0236d43bc68" Oct 29 23:41:33.538657 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:33.538622 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55"] Oct 29 23:41:39.289092 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:39.289052 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6"] Oct 29 23:41:39.319615 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:39.319579 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k"] Oct 29 23:41:42.656754 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:42.656713 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx"] Oct 29 23:41:42.666271 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:42.666239 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e165bd-mmtsx"] Oct 29 23:41:42.751960 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:42.751924 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66168d46-aaea-4034-8236-4e9d6a35c4a3" path="/var/lib/kubelet/pods/66168d46-aaea-4034-8236-4e9d6a35c4a3/volumes" Oct 29 23:41:44.022520 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:44.022479 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:41:44.290127 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:44.290023 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" podUID="880367dc-1403-444f-8b2c-cf9d6cf2dbf1" containerName="payload-processing" containerID="cri-o://aec0e658f1f4b9d243682d0a5af3e0c6a200209936b1ec4824ece95f1f476ad9" gracePeriod=25 Oct 29 23:41:44.321055 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:44.321010 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" podUID="33b9b88a-d944-44bc-b044-d1c80fca0555" containerName="payload-pre-processing" containerID="cri-o://56bf39a3a3cb019e0efb92d3c2865fb8ba231057f5ae67c97293f1609b2928c3" gracePeriod=25 Oct 29 23:41:45.222524 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:41:45.222488 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:41:45.622705 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:45.622611 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" podUID="33b9b88a-d944-44bc-b044-d1c80fca0555" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:41:45.622705 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:45.622642 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" podUID="880367dc-1403-444f-8b2c-cf9d6cf2dbf1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:41:48.559265 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.559223 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" podUID="9a8bb1d9-aad2-48a8-b589-13ee90356eff" containerName="main" containerID="cri-o://4caef3f1fb765c40f172302167ed98589d2cd3f02e43ff891eccd9beefda8eee" gracePeriod=45 Oct 29 23:41:48.814349 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.814289 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:41:48.881673 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.881634 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-dshm\") pod \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " Oct 29 23:41:48.881900 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.881688 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a8bb1d9-aad2-48a8-b589-13ee90356eff-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8bb1d9-aad2-48a8-b589-13ee90356eff-tls-certs\") pod \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " Oct 29 23:41:48.881984 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.881929 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-tmp-dir\") pod \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " Oct 29 23:41:48.882053 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.882016 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-model-cache\") pod \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " Oct 29 23:41:48.882098 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.882081 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8bb1d9-aad2-48a8-b589-13ee90356eff-kube-api-access-pfhz4\" (UniqueName: \"kubernetes.io/projected/9a8bb1d9-aad2-48a8-b589-13ee90356eff-kube-api-access-pfhz4\") pod \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " Oct 29 23:41:48.882151 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.882130 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-home\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-home\") pod \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\" (UID: \"9a8bb1d9-aad2-48a8-b589-13ee90356eff\") " Oct 29 23:41:48.882429 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.882395 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-tmp-dir" pod "9a8bb1d9-aad2-48a8-b589-13ee90356eff" (UID: "9a8bb1d9-aad2-48a8-b589-13ee90356eff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:41:48.882711 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.882687 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-home" pod "9a8bb1d9-aad2-48a8-b589-13ee90356eff" (UID: "9a8bb1d9-aad2-48a8-b589-13ee90356eff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:41:48.882801 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.882701 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-model-cache" pod "9a8bb1d9-aad2-48a8-b589-13ee90356eff" (UID: "9a8bb1d9-aad2-48a8-b589-13ee90356eff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:41:48.883162 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.883141 6507 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-tmp-dir\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:48.883260 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.883166 6507 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-model-cache\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:48.883260 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.883179 6507 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-home\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:48.884358 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.884331 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-dshm" pod "9a8bb1d9-aad2-48a8-b589-13ee90356eff" (UID: "9a8bb1d9-aad2-48a8-b589-13ee90356eff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:41:48.884678 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.884655 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8bb1d9-aad2-48a8-b589-13ee90356eff-tls-certs" pod "9a8bb1d9-aad2-48a8-b589-13ee90356eff" (UID: "9a8bb1d9-aad2-48a8-b589-13ee90356eff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:41:48.884879 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.884859 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8bb1d9-aad2-48a8-b589-13ee90356eff-kube-api-access-pfhz4" pod "9a8bb1d9-aad2-48a8-b589-13ee90356eff" (UID: "9a8bb1d9-aad2-48a8-b589-13ee90356eff"). InnerVolumeSpecName "kube-api-access-pfhz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:41:48.983970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.983937 6507 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8bb1d9-aad2-48a8-b589-13ee90356eff-dshm\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:48.983970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.983966 6507 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8bb1d9-aad2-48a8-b589-13ee90356eff-tls-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:48.983970 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:48.983976 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfhz4\" (UniqueName: \"kubernetes.io/projected/9a8bb1d9-aad2-48a8-b589-13ee90356eff-kube-api-access-pfhz4\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:49.055097 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:49.055057 6507 generic.go:356] "Generic (PLEG): container finished" podID="9a8bb1d9-aad2-48a8-b589-13ee90356eff" containerID="4caef3f1fb765c40f172302167ed98589d2cd3f02e43ff891eccd9beefda8eee" exitCode=0 Oct 29 23:41:49.055245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:49.055106 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" Oct 29 23:41:49.055245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:49.055142 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" event={"ID":"9a8bb1d9-aad2-48a8-b589-13ee90356eff","Type":"ContainerDied","Data":"4caef3f1fb765c40f172302167ed98589d2cd3f02e43ff891eccd9beefda8eee"} Oct 29 23:41:49.055245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:49.055188 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" event={"ID":"9a8bb1d9-aad2-48a8-b589-13ee90356eff","Type":"ContainerDied","Data":"0019f9322e5c9ded7d998c2baeda525cdc5acd6de44e511e4d4f8154811115a3"} Oct 29 23:41:49.055245 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:49.055205 6507 scope.go:122] "RemoveContainer" containerID="4caef3f1fb765c40f172302167ed98589d2cd3f02e43ff891eccd9beefda8eee" Oct 29 23:41:49.065223 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:49.065200 6507 scope.go:122] "RemoveContainer" containerID="4caef3f1fb765c40f172302167ed98589d2cd3f02e43ff891eccd9beefda8eee" Oct 29 23:41:49.065519 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:41:49.065500 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4caef3f1fb765c40f172302167ed98589d2cd3f02e43ff891eccd9beefda8eee\": container with ID starting with 4caef3f1fb765c40f172302167ed98589d2cd3f02e43ff891eccd9beefda8eee not found: ID does not exist" containerID="4caef3f1fb765c40f172302167ed98589d2cd3f02e43ff891eccd9beefda8eee" Oct 29 23:41:49.065576 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:49.065528 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4caef3f1fb765c40f172302167ed98589d2cd3f02e43ff891eccd9beefda8eee"} err="failed to get container status \"4caef3f1fb765c40f172302167ed98589d2cd3f02e43ff891eccd9beefda8eee\": rpc error: code = NotFound desc = could not find container \"4caef3f1fb765c40f172302167ed98589d2cd3f02e43ff891eccd9beefda8eee\": container with ID starting with 4caef3f1fb765c40f172302167ed98589d2cd3f02e43ff891eccd9beefda8eee not found: ID does not exist" Oct 29 23:41:49.112228 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:49.112201 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55"] Oct 29 23:41:49.119352 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:49.119325 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55"] Oct 29 23:41:49.794160 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:49.794117 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-e165bd/simulator-single-main-e165bd-kserve-5464785447-zsp55" podUID="9a8bb1d9-aad2-48a8-b589-13ee90356eff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.145:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:41:50.753587 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:50.753556 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8bb1d9-aad2-48a8-b589-13ee90356eff" path="/var/lib/kubelet/pods/9a8bb1d9-aad2-48a8-b589-13ee90356eff/volumes" Oct 29 23:41:50.850616 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:50.850553 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm"] Oct 29 23:41:50.851489 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:50.851275 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" podUID="741be03f-82a0-4001-a343-84f8f8a028f5" containerName="istio-proxy" containerID="cri-o://f797b04c10c6dbb37d945ce8936fbfa97345ffea9ab3f5c54d2e05703c614b60" gracePeriod=30 Oct 29 23:41:53.280682 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:53.280649 6507 scope.go:122] "RemoveContainer" containerID="c283966f8589e4b6b4f10e070650a39a0f1141d16352216d461e5871c9f2ed36" Oct 29 23:41:53.291437 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:53.291410 6507 scope.go:122] "RemoveContainer" containerID="7ca5ac9fb1b747a7b219145a015ba340d3ca184d5a56b7a38b3b41069dac32f1" Oct 29 23:41:53.299279 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:53.299257 6507 scope.go:122] "RemoveContainer" containerID="8bc0ef02dce3b1968c61ed3941f6c2ad0a0b0a8b1a6839fcc229bc3b8b3371dd" Oct 29 23:41:55.423984 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.423949 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv"] Oct 29 23:41:55.424427 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.424379 6507 cpu_manager.go:414] "containerMap: removing container" podUID="9a8bb1d9-aad2-48a8-b589-13ee90356eff" containerName="main" Oct 29 23:41:55.424427 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.424390 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8bb1d9-aad2-48a8-b589-13ee90356eff" containerName="main" Oct 29 23:41:55.424427 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.424405 6507 cpu_manager.go:414] "containerMap: removing container" podUID="66168d46-aaea-4034-8236-4e9d6a35c4a3" containerName="revoke-keys" Oct 29 23:41:55.424427 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.424412 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66168d46-aaea-4034-8236-4e9d6a35c4a3" containerName="revoke-keys" Oct 29 23:41:55.424631 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.424504 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8bb1d9-aad2-48a8-b589-13ee90356eff" containerName="main" Oct 29 23:41:55.424631 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.424516 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="66168d46-aaea-4034-8236-4e9d6a35c4a3" containerName="revoke-keys" Oct 29 23:41:55.429688 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.429663 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.454882 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.454850 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-809a6f59-kserve-self-signed-certs\"" Oct 29 23:41:55.462492 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.462443 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv"] Oct 29 23:41:55.554196 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.554159 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scj2p\" (UniqueName: \"kubernetes.io/projected/55a6367d-6371-4e1a-95b9-4a2e129c0399-kube-api-access-scj2p\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.554196 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.554201 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-home\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.554410 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.554225 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-tmp-dir\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.554410 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.554274 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a6367d-6371-4e1a-95b9-4a2e129c0399-tls-certs\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.554410 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.554379 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-model-cache\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.554410 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.554397 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-dshm\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.630211 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.630171 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" podUID="33b9b88a-d944-44bc-b044-d1c80fca0555" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:41:55.641491 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.641418 6507 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" podUID="880367dc-1403-444f-8b2c-cf9d6cf2dbf1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:41:55.655394 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.655363 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-tmp-dir\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.655599 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.655400 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a6367d-6371-4e1a-95b9-4a2e129c0399-tls-certs\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.655599 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.655423 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-model-cache\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.655599 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.655442 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-dshm\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.655599 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.655565 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scj2p\" (UniqueName: \"kubernetes.io/projected/55a6367d-6371-4e1a-95b9-4a2e129c0399-kube-api-access-scj2p\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.655599 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.655587 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-home\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.655907 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.655882 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-model-cache\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.655987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.655919 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-home\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.656049 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.656032 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-tmp-dir\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.657852 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.657825 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-dshm\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.657971 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.657953 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a6367d-6371-4e1a-95b9-4a2e129c0399-tls-certs\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.673183 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.673159 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scj2p\" (UniqueName: \"kubernetes.io/projected/55a6367d-6371-4e1a-95b9-4a2e129c0399-kube-api-access-scj2p\") pod \"e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:55.741807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:55.741768 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:56.040451 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.040410 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv"] Oct 29 23:41:56.044825 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:41:56.044784 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55a6367d_6371_4e1a_95b9_4a2e129c0399.slice/crio-a4de6abd94e444abc6f3a466d3cef6c99f45c52e3ab4946b29e764d28dd42edf WatchSource:0}: Error finding container a4de6abd94e444abc6f3a466d3cef6c99f45c52e3ab4946b29e764d28dd42edf: Status 404 returned error can't find the container with id a4de6abd94e444abc6f3a466d3cef6c99f45c52e3ab4946b29e764d28dd42edf Oct 29 23:41:56.092996 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.092952 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" event={"ID":"55a6367d-6371-4e1a-95b9-4a2e129c0399","Type":"ContainerStarted","Data":"a4de6abd94e444abc6f3a466d3cef6c99f45c52e3ab4946b29e764d28dd42edf"} Oct 29 23:41:56.094853 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.094827 6507 generic.go:356] "Generic (PLEG): container finished" podID="741be03f-82a0-4001-a343-84f8f8a028f5" containerID="f797b04c10c6dbb37d945ce8936fbfa97345ffea9ab3f5c54d2e05703c614b60" exitCode=0 Oct 29 23:41:56.094971 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.094875 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" event={"ID":"741be03f-82a0-4001-a343-84f8f8a028f5","Type":"ContainerDied","Data":"f797b04c10c6dbb37d945ce8936fbfa97345ffea9ab3f5c54d2e05703c614b60"} Oct 29 23:41:56.094971 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.094921 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" event={"ID":"741be03f-82a0-4001-a343-84f8f8a028f5","Type":"ContainerDied","Data":"b38da56d83d4173150d26739db241206e7aea78211e61bd4f4236de9de11fdf5"} Oct 29 23:41:56.094971 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.094940 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b38da56d83d4173150d26739db241206e7aea78211e61bd4f4236de9de11fdf5" Oct 29 23:41:56.094971 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.094953 6507 scope.go:122] "RemoveContainer" containerID="cdfa415defc0087512005144d2f60fd326ed0e32c6cfa840c7a47c091d23c2a3" Oct 29 23:41:56.096752 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.096731 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:41:56.161333 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.161302 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-workload-certs\") pod \"741be03f-82a0-4001-a343-84f8f8a028f5\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " Oct 29 23:41:56.161542 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.161369 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/741be03f-82a0-4001-a343-84f8f8a028f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741be03f-82a0-4001-a343-84f8f8a028f5-istio-podinfo\") pod \"741be03f-82a0-4001-a343-84f8f8a028f5\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " Oct 29 23:41:56.161542 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.161408 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741be03f-82a0-4001-a343-84f8f8a028f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741be03f-82a0-4001-a343-84f8f8a028f5-istiod-ca-cert\") pod \"741be03f-82a0-4001-a343-84f8f8a028f5\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " Oct 29 23:41:56.161542 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.161445 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741be03f-82a0-4001-a343-84f8f8a028f5-istio-token\" (UniqueName: \"kubernetes.io/projected/741be03f-82a0-4001-a343-84f8f8a028f5-istio-token\") pod \"741be03f-82a0-4001-a343-84f8f8a028f5\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " Oct 29 23:41:56.161542 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.161514 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-istio-envoy\") pod \"741be03f-82a0-4001-a343-84f8f8a028f5\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " Oct 29 23:41:56.161806 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.161580 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-workload-socket\") pod \"741be03f-82a0-4001-a343-84f8f8a028f5\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " Oct 29 23:41:56.161806 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.161591 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-workload-certs" pod "741be03f-82a0-4001-a343-84f8f8a028f5" (UID: "741be03f-82a0-4001-a343-84f8f8a028f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:41:56.161806 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.161660 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741be03f-82a0-4001-a343-84f8f8a028f5-kube-api-access-plt8n\" (UniqueName: \"kubernetes.io/projected/741be03f-82a0-4001-a343-84f8f8a028f5-kube-api-access-plt8n\") pod \"741be03f-82a0-4001-a343-84f8f8a028f5\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " Oct 29 23:41:56.161806 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.161732 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-istio-data\") pod \"741be03f-82a0-4001-a343-84f8f8a028f5\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " Oct 29 23:41:56.161806 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.161761 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-credential-socket\") pod \"741be03f-82a0-4001-a343-84f8f8a028f5\" (UID: \"741be03f-82a0-4001-a343-84f8f8a028f5\") " Oct 29 23:41:56.162074 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.161843 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741be03f-82a0-4001-a343-84f8f8a028f5-istiod-ca-cert" pod "741be03f-82a0-4001-a343-84f8f8a028f5" (UID: "741be03f-82a0-4001-a343-84f8f8a028f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:41:56.162149 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.162127 6507 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-workload-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:56.162216 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.162156 6507 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741be03f-82a0-4001-a343-84f8f8a028f5-istiod-ca-cert\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:56.162292 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.162270 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-workload-socket" pod "741be03f-82a0-4001-a343-84f8f8a028f5" (UID: "741be03f-82a0-4001-a343-84f8f8a028f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:41:56.162391 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.162346 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-istio-data" pod "741be03f-82a0-4001-a343-84f8f8a028f5" (UID: "741be03f-82a0-4001-a343-84f8f8a028f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:41:56.162391 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.162367 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-credential-socket" pod "741be03f-82a0-4001-a343-84f8f8a028f5" (UID: "741be03f-82a0-4001-a343-84f8f8a028f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:41:56.163987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.163960 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-istio-envoy" pod "741be03f-82a0-4001-a343-84f8f8a028f5" (UID: "741be03f-82a0-4001-a343-84f8f8a028f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:41:56.163987 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.163980 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/741be03f-82a0-4001-a343-84f8f8a028f5-istio-podinfo" pod "741be03f-82a0-4001-a343-84f8f8a028f5" (UID: "741be03f-82a0-4001-a343-84f8f8a028f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:41:56.164198 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.164179 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741be03f-82a0-4001-a343-84f8f8a028f5-istio-token" pod "741be03f-82a0-4001-a343-84f8f8a028f5" (UID: "741be03f-82a0-4001-a343-84f8f8a028f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:41:56.164398 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.164383 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741be03f-82a0-4001-a343-84f8f8a028f5-kube-api-access-plt8n" pod "741be03f-82a0-4001-a343-84f8f8a028f5" (UID: "741be03f-82a0-4001-a343-84f8f8a028f5"). InnerVolumeSpecName "kube-api-access-plt8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:41:56.262928 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.262890 6507 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-istio-envoy\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:56.262928 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.262920 6507 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-workload-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:56.262928 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.262931 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plt8n\" (UniqueName: \"kubernetes.io/projected/741be03f-82a0-4001-a343-84f8f8a028f5-kube-api-access-plt8n\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:56.263164 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.262943 6507 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-istio-data\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:56.263164 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.262951 6507 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741be03f-82a0-4001-a343-84f8f8a028f5-credential-socket\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:56.263164 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.262959 6507 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741be03f-82a0-4001-a343-84f8f8a028f5-istio-podinfo\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:56.263164 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:56.262968 6507 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741be03f-82a0-4001-a343-84f8f8a028f5-istio-token\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:41:57.101620 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:57.101566 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" event={"ID":"55a6367d-6371-4e1a-95b9-4a2e129c0399","Type":"ContainerStarted","Data":"50316a0d9917f9f9645d44eaa838a7092644a53baf4e4261154698a3d7f74e28"} Oct 29 23:41:57.102962 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:57.102936 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm" Oct 29 23:41:57.140641 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:57.140591 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" podStartSLOduration=2.1405766059999998 podStartE2EDuration="2.140576606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:41:57.1396105 +0000 UTC m=+1686.502615224" watchObservedRunningTime="2026-10-29 23:41:57.140576606 +0000 UTC m=+1686.503581323" Oct 29 23:41:57.202090 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:57.202054 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm"] Oct 29 23:41:57.207297 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:57.207263 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e165bd-openshift-default-6599955c97m2thm"] Oct 29 23:41:57.405645 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:57.405414 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:41:57.742388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:57.742356 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:41:58.755424 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:41:58.755381 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741be03f-82a0-4001-a343-84f8f8a028f5" path="/var/lib/kubelet/pods/741be03f-82a0-4001-a343-84f8f8a028f5/volumes" Oct 29 23:42:04.575862 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:04.575803 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" Oct 29 23:42:04.593591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:04.593566 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880367dc-1403-444f-8b2c-cf9d6cf2dbf1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880367dc-1403-444f-8b2c-cf9d6cf2dbf1-plugins-config-volume\") pod \"880367dc-1403-444f-8b2c-cf9d6cf2dbf1\" (UID: \"880367dc-1403-444f-8b2c-cf9d6cf2dbf1\") " Oct 29 23:42:04.593710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:04.593693 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880367dc-1403-444f-8b2c-cf9d6cf2dbf1-kube-api-access-75558\" (UniqueName: \"kubernetes.io/projected/880367dc-1403-444f-8b2c-cf9d6cf2dbf1-kube-api-access-75558\") pod \"880367dc-1403-444f-8b2c-cf9d6cf2dbf1\" (UID: \"880367dc-1403-444f-8b2c-cf9d6cf2dbf1\") " Oct 29 23:42:04.593885 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:04.593862 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880367dc-1403-444f-8b2c-cf9d6cf2dbf1-plugins-config-volume" pod "880367dc-1403-444f-8b2c-cf9d6cf2dbf1" (UID: "880367dc-1403-444f-8b2c-cf9d6cf2dbf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:42:04.594005 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:04.593992 6507 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880367dc-1403-444f-8b2c-cf9d6cf2dbf1-plugins-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:42:04.595752 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:04.595718 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880367dc-1403-444f-8b2c-cf9d6cf2dbf1-kube-api-access-75558" pod "880367dc-1403-444f-8b2c-cf9d6cf2dbf1" (UID: "880367dc-1403-444f-8b2c-cf9d6cf2dbf1"). InnerVolumeSpecName "kube-api-access-75558". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:42:04.605607 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:04.605586 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" Oct 29 23:42:04.694660 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:04.694627 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b9b88a-d944-44bc-b044-d1c80fca0555-kube-api-access-ssrnv\" (UniqueName: \"kubernetes.io/projected/33b9b88a-d944-44bc-b044-d1c80fca0555-kube-api-access-ssrnv\") pod \"33b9b88a-d944-44bc-b044-d1c80fca0555\" (UID: \"33b9b88a-d944-44bc-b044-d1c80fca0555\") " Oct 29 23:42:04.694871 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:04.694850 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b9b88a-d944-44bc-b044-d1c80fca0555-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9b88a-d944-44bc-b044-d1c80fca0555-plugins-config-volume\") pod \"33b9b88a-d944-44bc-b044-d1c80fca0555\" (UID: \"33b9b88a-d944-44bc-b044-d1c80fca0555\") " Oct 29 23:42:04.695114 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:04.695097 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75558\" (UniqueName: \"kubernetes.io/projected/880367dc-1403-444f-8b2c-cf9d6cf2dbf1-kube-api-access-75558\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:42:04.695191 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:04.695174 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b9b88a-d944-44bc-b044-d1c80fca0555-plugins-config-volume" pod "33b9b88a-d944-44bc-b044-d1c80fca0555" (UID: "33b9b88a-d944-44bc-b044-d1c80fca0555"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:42:04.696824 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:04.696801 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b9b88a-d944-44bc-b044-d1c80fca0555-kube-api-access-ssrnv" pod "33b9b88a-d944-44bc-b044-d1c80fca0555" (UID: "33b9b88a-d944-44bc-b044-d1c80fca0555"). InnerVolumeSpecName "kube-api-access-ssrnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:42:04.796007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:04.795967 6507 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9b88a-d944-44bc-b044-d1c80fca0555-plugins-config-volume\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:42:04.796007 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:04.795993 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssrnv\" (UniqueName: \"kubernetes.io/projected/33b9b88a-d944-44bc-b044-d1c80fca0555-kube-api-access-ssrnv\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:42:05.182106 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.182069 6507 generic.go:356] "Generic (PLEG): container finished" podID="880367dc-1403-444f-8b2c-cf9d6cf2dbf1" containerID="aec0e658f1f4b9d243682d0a5af3e0c6a200209936b1ec4824ece95f1f476ad9" exitCode=0 Oct 29 23:42:05.182304 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.182114 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" Oct 29 23:42:05.182304 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.182150 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" event={"ID":"880367dc-1403-444f-8b2c-cf9d6cf2dbf1","Type":"ContainerDied","Data":"aec0e658f1f4b9d243682d0a5af3e0c6a200209936b1ec4824ece95f1f476ad9"} Oct 29 23:42:05.182304 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.182204 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6" event={"ID":"880367dc-1403-444f-8b2c-cf9d6cf2dbf1","Type":"ContainerDied","Data":"5dd8d909e629b3710908ad4f920de7ca436b2a4597079c001816954d47db6c8e"} Oct 29 23:42:05.182304 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.182226 6507 scope.go:122] "RemoveContainer" containerID="aec0e658f1f4b9d243682d0a5af3e0c6a200209936b1ec4824ece95f1f476ad9" Oct 29 23:42:05.183661 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.183638 6507 generic.go:356] "Generic (PLEG): container finished" podID="33b9b88a-d944-44bc-b044-d1c80fca0555" containerID="56bf39a3a3cb019e0efb92d3c2865fb8ba231057f5ae67c97293f1609b2928c3" exitCode=0 Oct 29 23:42:05.183783 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.183666 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" event={"ID":"33b9b88a-d944-44bc-b044-d1c80fca0555","Type":"ContainerDied","Data":"56bf39a3a3cb019e0efb92d3c2865fb8ba231057f5ae67c97293f1609b2928c3"} Oct 29 23:42:05.183783 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.183695 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" Oct 29 23:42:05.183783 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.183716 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k" event={"ID":"33b9b88a-d944-44bc-b044-d1c80fca0555","Type":"ContainerDied","Data":"cf83beb120743bb6fe67d6b9e47bb9b4309dc292994b8f10bf5a80225be9d062"} Oct 29 23:42:05.190963 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.190944 6507 scope.go:122] "RemoveContainer" containerID="aec0e658f1f4b9d243682d0a5af3e0c6a200209936b1ec4824ece95f1f476ad9" Oct 29 23:42:05.191247 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:42:05.191227 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec0e658f1f4b9d243682d0a5af3e0c6a200209936b1ec4824ece95f1f476ad9\": container with ID starting with aec0e658f1f4b9d243682d0a5af3e0c6a200209936b1ec4824ece95f1f476ad9 not found: ID does not exist" containerID="aec0e658f1f4b9d243682d0a5af3e0c6a200209936b1ec4824ece95f1f476ad9" Oct 29 23:42:05.191300 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.191256 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec0e658f1f4b9d243682d0a5af3e0c6a200209936b1ec4824ece95f1f476ad9"} err="failed to get container status \"aec0e658f1f4b9d243682d0a5af3e0c6a200209936b1ec4824ece95f1f476ad9\": rpc error: code = NotFound desc = could not find container \"aec0e658f1f4b9d243682d0a5af3e0c6a200209936b1ec4824ece95f1f476ad9\": container with ID starting with aec0e658f1f4b9d243682d0a5af3e0c6a200209936b1ec4824ece95f1f476ad9 not found: ID does not exist" Oct 29 23:42:05.191300 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.191272 6507 scope.go:122] "RemoveContainer" containerID="56bf39a3a3cb019e0efb92d3c2865fb8ba231057f5ae67c97293f1609b2928c3" Oct 29 23:42:05.199157 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.199138 6507 scope.go:122] "RemoveContainer" containerID="56bf39a3a3cb019e0efb92d3c2865fb8ba231057f5ae67c97293f1609b2928c3" Oct 29 23:42:05.199432 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:42:05.199414 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bf39a3a3cb019e0efb92d3c2865fb8ba231057f5ae67c97293f1609b2928c3\": container with ID starting with 56bf39a3a3cb019e0efb92d3c2865fb8ba231057f5ae67c97293f1609b2928c3 not found: ID does not exist" containerID="56bf39a3a3cb019e0efb92d3c2865fb8ba231057f5ae67c97293f1609b2928c3" Oct 29 23:42:05.199514 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.199441 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bf39a3a3cb019e0efb92d3c2865fb8ba231057f5ae67c97293f1609b2928c3"} err="failed to get container status \"56bf39a3a3cb019e0efb92d3c2865fb8ba231057f5ae67c97293f1609b2928c3\": rpc error: code = NotFound desc = could not find container \"56bf39a3a3cb019e0efb92d3c2865fb8ba231057f5ae67c97293f1609b2928c3\": container with ID starting with 56bf39a3a3cb019e0efb92d3c2865fb8ba231057f5ae67c97293f1609b2928c3 not found: ID does not exist" Oct 29 23:42:05.223959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.223928 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6"] Oct 29 23:42:05.233392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.233366 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e165bd-db74b6bf4gd8l6"] Oct 29 23:42:05.262369 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.262339 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k"] Oct 29 23:42:05.269952 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.269922 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e165bd-56745pq54k"] Oct 29 23:42:05.742523 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.742489 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:42:05.755399 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:05.755367 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:42:06.201854 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:06.201819 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:42:06.751368 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:06.751334 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b9b88a-d944-44bc-b044-d1c80fca0555" path="/var/lib/kubelet/pods/33b9b88a-d944-44bc-b044-d1c80fca0555/volumes" Oct 29 23:42:06.751777 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:06.751743 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="880367dc-1403-444f-8b2c-cf9d6cf2dbf1" path="/var/lib/kubelet/pods/880367dc-1403-444f-8b2c-cf9d6cf2dbf1/volumes" Oct 29 23:42:15.223268 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:42:15.223227 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:42:16.975440 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:16.975404 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4"] Oct 29 23:42:16.976013 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:16.975992 6507 cpu_manager.go:414] "containerMap: removing container" podUID="741be03f-82a0-4001-a343-84f8f8a028f5" containerName="istio-proxy" Oct 29 23:42:16.976113 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:16.976015 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741be03f-82a0-4001-a343-84f8f8a028f5" containerName="istio-proxy" Oct 29 23:42:16.976113 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:16.976059 6507 cpu_manager.go:414] "containerMap: removing container" podUID="880367dc-1403-444f-8b2c-cf9d6cf2dbf1" containerName="payload-processing" Oct 29 23:42:16.976113 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:16.976067 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880367dc-1403-444f-8b2c-cf9d6cf2dbf1" containerName="payload-processing" Oct 29 23:42:16.976113 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:16.976085 6507 cpu_manager.go:414] "containerMap: removing container" podUID="33b9b88a-d944-44bc-b044-d1c80fca0555" containerName="payload-pre-processing" Oct 29 23:42:16.976113 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:16.976094 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b9b88a-d944-44bc-b044-d1c80fca0555" containerName="payload-pre-processing" Oct 29 23:42:16.976365 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:16.976127 6507 cpu_manager.go:414] "containerMap: removing container" podUID="741be03f-82a0-4001-a343-84f8f8a028f5" containerName="istio-proxy" Oct 29 23:42:16.976365 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:16.976135 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741be03f-82a0-4001-a343-84f8f8a028f5" containerName="istio-proxy" Oct 29 23:42:16.976365 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:16.976262 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="741be03f-82a0-4001-a343-84f8f8a028f5" containerName="istio-proxy" Oct 29 23:42:16.976365 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:16.976280 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="741be03f-82a0-4001-a343-84f8f8a028f5" containerName="istio-proxy" Oct 29 23:42:16.976365 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:16.976291 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="880367dc-1403-444f-8b2c-cf9d6cf2dbf1" containerName="payload-processing" Oct 29 23:42:16.976365 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:16.976303 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b9b88a-d944-44bc-b044-d1c80fca0555" containerName="payload-pre-processing" Oct 29 23:42:16.979566 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:16.979543 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.003291 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.003243 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-809a6f59-kserve-self-signed-certs\"" Oct 29 23:42:17.005297 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.005272 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4"] Oct 29 23:42:17.018250 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.018198 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-home\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.018454 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.018360 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-model-cache\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.018454 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.018474 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-tls-certs\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.018605 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.018557 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhh62\" (UniqueName: \"kubernetes.io/projected/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-kube-api-access-jhh62\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.018605 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.018595 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-dshm\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.018710 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.018613 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-tmp-dir\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.119976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.119938 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-dshm\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.119976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.119973 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-tmp-dir\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.120237 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.120078 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-home\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.120237 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.120125 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-model-cache\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.120237 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.120166 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-tls-certs\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.120237 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.120220 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhh62\" (UniqueName: \"kubernetes.io/projected/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-kube-api-access-jhh62\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.120534 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.120510 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-tmp-dir\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.120600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.120543 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-model-cache\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.120600 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.120584 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-home\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.122213 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.122189 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-dshm\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.122647 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.122630 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-tls-certs\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.142807 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.142782 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhh62\" (UniqueName: \"kubernetes.io/projected/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-kube-api-access-jhh62\") pod \"e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.296373 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.296288 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:17.594370 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:17.594285 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4"] Oct 29 23:42:17.597806 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:42:17.597776 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28596a2a_5fa3_47f3_bd47_1f68ee8e3fec.slice/crio-1208650a4eb4e983bc3bc9289069c72757b1de6a2cf4594140140e5fd703d9b1 WatchSource:0}: Error finding container 1208650a4eb4e983bc3bc9289069c72757b1de6a2cf4594140140e5fd703d9b1: Status 404 returned error can't find the container with id 1208650a4eb4e983bc3bc9289069c72757b1de6a2cf4594140140e5fd703d9b1 Oct 29 23:42:18.256266 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:18.256227 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" event={"ID":"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec","Type":"ContainerStarted","Data":"6910ddef44e0074579cfe27872a05bc54328fbdfcba6ae8535a51a0c984eddc5"} Oct 29 23:42:18.256266 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:18.256272 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" event={"ID":"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec","Type":"ContainerStarted","Data":"1208650a4eb4e983bc3bc9289069c72757b1de6a2cf4594140140e5fd703d9b1"} Oct 29 23:42:18.291355 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:18.291300 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" podStartSLOduration=2.291288096 podStartE2EDuration="2.291288096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:42:18.290829243 +0000 UTC m=+1707.653833979" watchObservedRunningTime="2026-10-29 23:42:18.291288096 +0000 UTC m=+1707.654292819" Oct 29 23:42:18.297201 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:18.297175 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:19.307977 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:19.307942 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:42:27.297191 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:27.297153 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:27.310428 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:27.310403 6507 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:27.390583 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:27.390556 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:42:35.041872 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:35.041789 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:42:36.195774 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.195735 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6"] Oct 29 23:42:36.202001 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.201965 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" Oct 29 23:42:36.217957 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.217927 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:42:36.227516 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.227484 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:42:36.227797 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.227779 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:42:36.228008 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.227991 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:42:36.238492 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.230320 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6"] Oct 29 23:42:36.373162 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.373105 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e78642-43a0-4069-9c41-9ce4126fd145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-tw6r6\" (UID: \"56e78642-43a0-4069-9c41-9ce4126fd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" Oct 29 23:42:36.373388 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.373185 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e78642-43a0-4069-9c41-9ce4126fd145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-tw6r6\" (UID: \"56e78642-43a0-4069-9c41-9ce4126fd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" Oct 29 23:42:36.373490 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.373436 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4spc\" (UniqueName: \"kubernetes.io/projected/56e78642-43a0-4069-9c41-9ce4126fd145-kube-api-access-z4spc\") pod \"maas-api-revoke-sub-452b6d7b-tw6r6\" (UID: \"56e78642-43a0-4069-9c41-9ce4126fd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" Oct 29 23:42:36.474999 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.474887 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e78642-43a0-4069-9c41-9ce4126fd145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-tw6r6\" (UID: \"56e78642-43a0-4069-9c41-9ce4126fd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" Oct 29 23:42:36.475203 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.475067 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4spc\" (UniqueName: \"kubernetes.io/projected/56e78642-43a0-4069-9c41-9ce4126fd145-kube-api-access-z4spc\") pod \"maas-api-revoke-sub-452b6d7b-tw6r6\" (UID: \"56e78642-43a0-4069-9c41-9ce4126fd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" Oct 29 23:42:36.475203 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.475175 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e78642-43a0-4069-9c41-9ce4126fd145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-tw6r6\" (UID: \"56e78642-43a0-4069-9c41-9ce4126fd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" Oct 29 23:42:36.476080 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.476043 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e78642-43a0-4069-9c41-9ce4126fd145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-tw6r6\" (UID: \"56e78642-43a0-4069-9c41-9ce4126fd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" Oct 29 23:42:36.478447 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.478409 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e78642-43a0-4069-9c41-9ce4126fd145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-tw6r6\" (UID: \"56e78642-43a0-4069-9c41-9ce4126fd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" Oct 29 23:42:36.493351 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.493306 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4spc\" (UniqueName: \"kubernetes.io/projected/56e78642-43a0-4069-9c41-9ce4126fd145-kube-api-access-z4spc\") pod \"maas-api-revoke-sub-452b6d7b-tw6r6\" (UID: \"56e78642-43a0-4069-9c41-9ce4126fd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" Oct 29 23:42:36.521959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:36.521900 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" Oct 29 23:42:37.157749 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:37.157715 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6"] Oct 29 23:42:37.501500 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:37.483201 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" event={"ID":"56e78642-43a0-4069-9c41-9ce4126fd145","Type":"ContainerStarted","Data":"34b96266abce2a7b4e45018cf9c1f81154003f6d5863e38bc945fd0147ac1091"} Oct 29 23:42:38.519925 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:38.518896 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" event={"ID":"56e78642-43a0-4069-9c41-9ce4126fd145","Type":"ContainerStarted","Data":"285ffd7828cc097a1fc6a128602a79ffea02871d3705c195ae4072da32da2572"} Oct 29 23:42:38.574285 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:38.571563 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" podStartSLOduration=2.5715446699999998 podStartE2EDuration="2.57154467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:42:38.570127016 +0000 UTC m=+1727.933131739" watchObservedRunningTime="2026-10-29 23:42:38.57154467 +0000 UTC m=+1727.934549394" Oct 29 23:42:39.525017 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:39.524985 6507 generic.go:356] "Generic (PLEG): container finished" podID="56e78642-43a0-4069-9c41-9ce4126fd145" containerID="285ffd7828cc097a1fc6a128602a79ffea02871d3705c195ae4072da32da2572" exitCode=0 Oct 29 23:42:39.525510 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:39.525047 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" event={"ID":"56e78642-43a0-4069-9c41-9ce4126fd145","Type":"ContainerDied","Data":"285ffd7828cc097a1fc6a128602a79ffea02871d3705c195ae4072da32da2572"} Oct 29 23:42:40.669247 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:40.669219 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" Oct 29 23:42:40.756544 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:40.756512 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e78642-43a0-4069-9c41-9ce4126fd145-kube-api-access-z4spc\" (UniqueName: \"kubernetes.io/projected/56e78642-43a0-4069-9c41-9ce4126fd145-kube-api-access-z4spc\") pod \"56e78642-43a0-4069-9c41-9ce4126fd145\" (UID: \"56e78642-43a0-4069-9c41-9ce4126fd145\") " Oct 29 23:42:40.756707 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:40.756552 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56e78642-43a0-4069-9c41-9ce4126fd145-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e78642-43a0-4069-9c41-9ce4126fd145-maas-api-openshift-ca\") pod \"56e78642-43a0-4069-9c41-9ce4126fd145\" (UID: \"56e78642-43a0-4069-9c41-9ce4126fd145\") " Oct 29 23:42:40.756749 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:40.756712 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56e78642-43a0-4069-9c41-9ce4126fd145-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e78642-43a0-4069-9c41-9ce4126fd145-maas-api-xks-ca\") pod \"56e78642-43a0-4069-9c41-9ce4126fd145\" (UID: \"56e78642-43a0-4069-9c41-9ce4126fd145\") " Oct 29 23:42:40.757691 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:40.756915 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e78642-43a0-4069-9c41-9ce4126fd145-maas-api-openshift-ca" pod "56e78642-43a0-4069-9c41-9ce4126fd145" (UID: "56e78642-43a0-4069-9c41-9ce4126fd145"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:42:40.757691 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:40.757370 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e78642-43a0-4069-9c41-9ce4126fd145-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:42:40.758903 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:40.758876 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56e78642-43a0-4069-9c41-9ce4126fd145-maas-api-xks-ca" pod "56e78642-43a0-4069-9c41-9ce4126fd145" (UID: "56e78642-43a0-4069-9c41-9ce4126fd145"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:42:40.759024 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:40.758934 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e78642-43a0-4069-9c41-9ce4126fd145-kube-api-access-z4spc" pod "56e78642-43a0-4069-9c41-9ce4126fd145" (UID: "56e78642-43a0-4069-9c41-9ce4126fd145"). InnerVolumeSpecName "kube-api-access-z4spc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:42:40.858431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:40.858345 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e78642-43a0-4069-9c41-9ce4126fd145-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:42:40.858431 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:40.858380 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4spc\" (UniqueName: \"kubernetes.io/projected/56e78642-43a0-4069-9c41-9ce4126fd145-kube-api-access-z4spc\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:42:41.536703 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:41.536675 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" Oct 29 23:42:41.536880 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:41.536668 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-tw6r6" event={"ID":"56e78642-43a0-4069-9c41-9ce4126fd145","Type":"ContainerDied","Data":"34b96266abce2a7b4e45018cf9c1f81154003f6d5863e38bc945fd0147ac1091"} Oct 29 23:42:41.536880 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:41.536801 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34b96266abce2a7b4e45018cf9c1f81154003f6d5863e38bc945fd0147ac1091" Oct 29 23:42:45.223154 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:42:45.223105 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:42:47.431278 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:47.431240 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv"] Oct 29 23:42:49.214527 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:49.214485 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:42:49.998478 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:49.998423 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4"] Oct 29 23:42:53.331028 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.330976 6507 scope.go:122] "RemoveContainer" containerID="9e75cc8d32d4153178a7a10018d03b5ebb7834669119ad37cb4fa1d8f988362c" Oct 29 23:42:53.343830 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.343794 6507 scope.go:122] "RemoveContainer" containerID="596f2763005111626621801eb8a273516a0255858e659afbea6074afee47718f" Oct 29 23:42:53.356321 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.356292 6507 scope.go:122] "RemoveContainer" containerID="abbaf0f855fac1e04cc8b3217b6a4515ab5d3308ea6939e7e5b3cf09a6fd68cd" Oct 29 23:42:53.368575 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.368548 6507 scope.go:122] "RemoveContainer" containerID="30a5a71a6e24210c5b8df2eaf5ee9a531008f9922fe186ec9709378b1c4eeb5a" Oct 29 23:42:53.772976 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.772933 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8"] Oct 29 23:42:53.773611 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.773587 6507 cpu_manager.go:414] "containerMap: removing container" podUID="56e78642-43a0-4069-9c41-9ce4126fd145" containerName="revoke-keys" Oct 29 23:42:53.773741 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.773613 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e78642-43a0-4069-9c41-9ce4126fd145" containerName="revoke-keys" Oct 29 23:42:53.773741 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.773719 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e78642-43a0-4069-9c41-9ce4126fd145" containerName="revoke-keys" Oct 29 23:42:53.779267 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.779238 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8" Oct 29 23:42:53.794588 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.794545 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:42:53.794806 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.794790 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:42:53.806495 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.803823 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8"] Oct 29 23:42:53.806495 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.803940 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:42:53.806495 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.804023 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:42:53.941293 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.941229 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-qhdj8\" (UID: \"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8" Oct 29 23:42:53.941539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.941321 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf55c\" (UniqueName: \"kubernetes.io/projected/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-kube-api-access-kf55c\") pod \"maas-api-revoke-sub-3e31a506-qhdj8\" (UID: \"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8" Oct 29 23:42:53.941539 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:53.941383 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-qhdj8\" (UID: \"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8" Oct 29 23:42:54.042394 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:54.042289 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-qhdj8\" (UID: \"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8" Oct 29 23:42:54.042394 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:54.042358 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf55c\" (UniqueName: \"kubernetes.io/projected/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-kube-api-access-kf55c\") pod \"maas-api-revoke-sub-3e31a506-qhdj8\" (UID: \"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8" Oct 29 23:42:54.042670 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:54.042419 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-qhdj8\" (UID: \"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8" Oct 29 23:42:54.043342 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:54.043309 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-qhdj8\" (UID: \"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8" Oct 29 23:42:54.045496 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:54.045438 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-qhdj8\" (UID: \"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8" Oct 29 23:42:54.060425 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:54.060389 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf55c\" (UniqueName: \"kubernetes.io/projected/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-kube-api-access-kf55c\") pod \"maas-api-revoke-sub-3e31a506-qhdj8\" (UID: \"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8" Oct 29 23:42:54.099906 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:54.099623 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8" Oct 29 23:42:54.973483 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:54.972577 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8"] Oct 29 23:42:55.047493 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:55.046877 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96"] Oct 29 23:42:55.121831 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:55.118956 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881bcb76-9fs96"] Oct 29 23:42:55.679748 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:55.679647 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8" event={"ID":"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79","Type":"ContainerStarted","Data":"f31c8f4c52e3820f6eb8bdfc507c8bf35da24bcee989c986f41c4cd3a69b5c10"} Oct 29 23:42:56.087277 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:56.087215 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:42:56.685706 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:56.685676 6507 generic.go:356] "Generic (PLEG): container finished" podID="18d09d9c-b1d3-4d4d-a0a2-1acc70371a79" containerID="ee04c0e45f43e59d1e7a4624f94798d726204a7f0f81f371cb030fe35b254fd7" exitCode=0 Oct 29 23:42:56.685872 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:56.685758 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8" event={"ID":"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79","Type":"ContainerDied","Data":"ee04c0e45f43e59d1e7a4624f94798d726204a7f0f81f371cb030fe35b254fd7"} Oct 29 23:42:56.751784 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:56.751747 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adf96c14-1212-49c5-98e8-babd7362dc5a" path="/var/lib/kubelet/pods/adf96c14-1212-49c5-98e8-babd7362dc5a/volumes" Oct 29 23:42:57.828518 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:57.828487 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8" Oct 29 23:42:57.905100 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:57.905068 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-kube-api-access-kf55c\" (UniqueName: \"kubernetes.io/projected/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-kube-api-access-kf55c\") pod \"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79\" (UID: \"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79\") " Oct 29 23:42:57.905286 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:57.905153 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-maas-api-xks-ca\") pod \"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79\" (UID: \"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79\") " Oct 29 23:42:57.905286 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:57.905254 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-maas-api-openshift-ca\") pod \"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79\" (UID: \"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79\") " Oct 29 23:42:57.905634 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:57.905607 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-maas-api-openshift-ca" pod "18d09d9c-b1d3-4d4d-a0a2-1acc70371a79" (UID: "18d09d9c-b1d3-4d4d-a0a2-1acc70371a79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:42:57.907309 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:57.907283 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-maas-api-xks-ca" pod "18d09d9c-b1d3-4d4d-a0a2-1acc70371a79" (UID: "18d09d9c-b1d3-4d4d-a0a2-1acc70371a79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:42:57.907432 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:57.907360 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-kube-api-access-kf55c" pod "18d09d9c-b1d3-4d4d-a0a2-1acc70371a79" (UID: "18d09d9c-b1d3-4d4d-a0a2-1acc70371a79"). InnerVolumeSpecName "kube-api-access-kf55c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:42:58.006427 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:58.006342 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:42:58.006427 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:58.006373 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:42:58.006427 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:58.006383 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf55c\" (UniqueName: \"kubernetes.io/projected/18d09d9c-b1d3-4d4d-a0a2-1acc70371a79-kube-api-access-kf55c\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:42:58.698525 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:58.698491 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8" Oct 29 23:42:58.698725 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:58.698497 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qhdj8" event={"ID":"18d09d9c-b1d3-4d4d-a0a2-1acc70371a79","Type":"ContainerDied","Data":"f31c8f4c52e3820f6eb8bdfc507c8bf35da24bcee989c986f41c4cd3a69b5c10"} Oct 29 23:42:58.698725 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:42:58.698622 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f31c8f4c52e3820f6eb8bdfc507c8bf35da24bcee989c986f41c4cd3a69b5c10" Oct 29 23:43:02.451291 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.451248 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" podUID="55a6367d-6371-4e1a-95b9-4a2e129c0399" containerName="main" containerID="cri-o://50316a0d9917f9f9645d44eaa838a7092644a53baf4e4261154698a3d7f74e28" gracePeriod=45 Oct 29 23:43:02.715392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.715315 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:43:02.719744 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.719719 6507 generic.go:356] "Generic (PLEG): container finished" podID="55a6367d-6371-4e1a-95b9-4a2e129c0399" containerID="50316a0d9917f9f9645d44eaa838a7092644a53baf4e4261154698a3d7f74e28" exitCode=0 Oct 29 23:43:02.719877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.719765 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" Oct 29 23:43:02.719877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.719801 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" event={"ID":"55a6367d-6371-4e1a-95b9-4a2e129c0399","Type":"ContainerDied","Data":"50316a0d9917f9f9645d44eaa838a7092644a53baf4e4261154698a3d7f74e28"} Oct 29 23:43:02.719877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.719857 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv" event={"ID":"55a6367d-6371-4e1a-95b9-4a2e129c0399","Type":"ContainerDied","Data":"a4de6abd94e444abc6f3a466d3cef6c99f45c52e3ab4946b29e764d28dd42edf"} Oct 29 23:43:02.720043 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.719882 6507 scope.go:122] "RemoveContainer" containerID="50316a0d9917f9f9645d44eaa838a7092644a53baf4e4261154698a3d7f74e28" Oct 29 23:43:02.729257 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.729233 6507 scope.go:122] "RemoveContainer" containerID="50316a0d9917f9f9645d44eaa838a7092644a53baf4e4261154698a3d7f74e28" Oct 29 23:43:02.729613 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:43:02.729590 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50316a0d9917f9f9645d44eaa838a7092644a53baf4e4261154698a3d7f74e28\": container with ID starting with 50316a0d9917f9f9645d44eaa838a7092644a53baf4e4261154698a3d7f74e28 not found: ID does not exist" containerID="50316a0d9917f9f9645d44eaa838a7092644a53baf4e4261154698a3d7f74e28" Oct 29 23:43:02.729674 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.729624 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50316a0d9917f9f9645d44eaa838a7092644a53baf4e4261154698a3d7f74e28"} err="failed to get container status \"50316a0d9917f9f9645d44eaa838a7092644a53baf4e4261154698a3d7f74e28\": rpc error: code = NotFound desc = could not find container \"50316a0d9917f9f9645d44eaa838a7092644a53baf4e4261154698a3d7f74e28\": container with ID starting with 50316a0d9917f9f9645d44eaa838a7092644a53baf4e4261154698a3d7f74e28 not found: ID does not exist" Oct 29 23:43:02.855788 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.855744 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-dshm\") pod \"55a6367d-6371-4e1a-95b9-4a2e129c0399\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " Oct 29 23:43:02.856005 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.855807 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a6367d-6371-4e1a-95b9-4a2e129c0399-kube-api-access-scj2p\" (UniqueName: \"kubernetes.io/projected/55a6367d-6371-4e1a-95b9-4a2e129c0399-kube-api-access-scj2p\") pod \"55a6367d-6371-4e1a-95b9-4a2e129c0399\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " Oct 29 23:43:02.856005 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.855878 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-home\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-home\") pod \"55a6367d-6371-4e1a-95b9-4a2e129c0399\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " Oct 29 23:43:02.856005 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.855952 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55a6367d-6371-4e1a-95b9-4a2e129c0399-tls-certs\" (UniqueName: \"kubernetes.io/secret/55a6367d-6371-4e1a-95b9-4a2e129c0399-tls-certs\") pod \"55a6367d-6371-4e1a-95b9-4a2e129c0399\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " Oct 29 23:43:02.856168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.856004 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-model-cache\") pod \"55a6367d-6371-4e1a-95b9-4a2e129c0399\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " Oct 29 23:43:02.856168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.856072 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-tmp-dir\") pod \"55a6367d-6371-4e1a-95b9-4a2e129c0399\" (UID: \"55a6367d-6371-4e1a-95b9-4a2e129c0399\") " Oct 29 23:43:02.856168 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.856145 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-home" pod "55a6367d-6371-4e1a-95b9-4a2e129c0399" (UID: "55a6367d-6371-4e1a-95b9-4a2e129c0399"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:02.856309 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.856284 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-model-cache" pod "55a6367d-6371-4e1a-95b9-4a2e129c0399" (UID: "55a6367d-6371-4e1a-95b9-4a2e129c0399"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:02.856424 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.856390 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-tmp-dir" pod "55a6367d-6371-4e1a-95b9-4a2e129c0399" (UID: "55a6367d-6371-4e1a-95b9-4a2e129c0399"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:02.856624 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.856606 6507 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-model-cache\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:02.856711 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.856628 6507 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-tmp-dir\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:02.856711 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.856640 6507 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-home\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:02.858198 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.858168 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a6367d-6371-4e1a-95b9-4a2e129c0399-kube-api-access-scj2p" pod "55a6367d-6371-4e1a-95b9-4a2e129c0399" (UID: "55a6367d-6371-4e1a-95b9-4a2e129c0399"). InnerVolumeSpecName "kube-api-access-scj2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:43:02.858308 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.858202 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55a6367d-6371-4e1a-95b9-4a2e129c0399-tls-certs" pod "55a6367d-6371-4e1a-95b9-4a2e129c0399" (UID: "55a6367d-6371-4e1a-95b9-4a2e129c0399"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:43:02.858308 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.858227 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-dshm" pod "55a6367d-6371-4e1a-95b9-4a2e129c0399" (UID: "55a6367d-6371-4e1a-95b9-4a2e129c0399"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:02.958134 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.958098 6507 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a6367d-6371-4e1a-95b9-4a2e129c0399-tls-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:02.958134 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.958128 6507 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a6367d-6371-4e1a-95b9-4a2e129c0399-dshm\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:02.958134 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:02.958138 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scj2p\" (UniqueName: \"kubernetes.io/projected/55a6367d-6371-4e1a-95b9-4a2e129c0399-kube-api-access-scj2p\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:03.061559 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:03.061515 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv"] Oct 29 23:43:03.070203 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:03.070168 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-809a6f59-kserve-6d6dfcdcfc-m57rv"] Oct 29 23:43:04.752035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:04.751996 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a6367d-6371-4e1a-95b9-4a2e129c0399" path="/var/lib/kubelet/pods/55a6367d-6371-4e1a-95b9-4a2e129c0399/volumes" Oct 29 23:43:04.869402 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:04.869367 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:43:05.032231 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.032148 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" podUID="28596a2a-5fa3-47f3-bd47-1f68ee8e3fec" containerName="main" containerID="cri-o://6910ddef44e0074579cfe27872a05bc54328fbdfcba6ae8535a51a0c984eddc5" gracePeriod=45 Oct 29 23:43:05.314821 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.314789 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:43:05.503485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.502422 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-tls-certs\" (UniqueName: \"kubernetes.io/secret/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-tls-certs\") pod \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " Oct 29 23:43:05.503485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.502495 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-dshm\") pod \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " Oct 29 23:43:05.503485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.502576 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-tmp-dir\") pod \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " Oct 29 23:43:05.503485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.502622 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-kube-api-access-jhh62\" (UniqueName: \"kubernetes.io/projected/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-kube-api-access-jhh62\") pod \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " Oct 29 23:43:05.503485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.502674 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-home\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-home\") pod \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " Oct 29 23:43:05.503485 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.502748 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-model-cache\") pod \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\" (UID: \"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec\") " Oct 29 23:43:05.519644 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.519594 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-model-cache" pod "28596a2a-5fa3-47f3-bd47-1f68ee8e3fec" (UID: "28596a2a-5fa3-47f3-bd47-1f68ee8e3fec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:05.545240 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.545153 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-tmp-dir" pod "28596a2a-5fa3-47f3-bd47-1f68ee8e3fec" (UID: "28596a2a-5fa3-47f3-bd47-1f68ee8e3fec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:05.546180 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.546154 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-home" pod "28596a2a-5fa3-47f3-bd47-1f68ee8e3fec" (UID: "28596a2a-5fa3-47f3-bd47-1f68ee8e3fec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:05.572808 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.572768 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-tls-certs" pod "28596a2a-5fa3-47f3-bd47-1f68ee8e3fec" (UID: "28596a2a-5fa3-47f3-bd47-1f68ee8e3fec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:43:05.576352 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.576294 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-dshm" pod "28596a2a-5fa3-47f3-bd47-1f68ee8e3fec" (UID: "28596a2a-5fa3-47f3-bd47-1f68ee8e3fec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:43:05.582504 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.576649 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-kube-api-access-jhh62" pod "28596a2a-5fa3-47f3-bd47-1f68ee8e3fec" (UID: "28596a2a-5fa3-47f3-bd47-1f68ee8e3fec"). InnerVolumeSpecName "kube-api-access-jhh62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:43:05.604240 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.604095 6507 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-model-cache\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:05.604240 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.604142 6507 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-tls-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:05.604240 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.604157 6507 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-dshm\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:05.604240 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.604172 6507 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-tmp-dir\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:05.604240 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.604188 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhh62\" (UniqueName: \"kubernetes.io/projected/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-kube-api-access-jhh62\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:05.604240 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.604202 6507 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec-home\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:05.764510 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.759878 6507 generic.go:356] "Generic (PLEG): container finished" podID="28596a2a-5fa3-47f3-bd47-1f68ee8e3fec" containerID="6910ddef44e0074579cfe27872a05bc54328fbdfcba6ae8535a51a0c984eddc5" exitCode=0 Oct 29 23:43:05.764510 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.760034 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" event={"ID":"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec","Type":"ContainerDied","Data":"6910ddef44e0074579cfe27872a05bc54328fbdfcba6ae8535a51a0c984eddc5"} Oct 29 23:43:05.764510 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.760073 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" event={"ID":"28596a2a-5fa3-47f3-bd47-1f68ee8e3fec","Type":"ContainerDied","Data":"1208650a4eb4e983bc3bc9289069c72757b1de6a2cf4594140140e5fd703d9b1"} Oct 29 23:43:05.764510 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.760099 6507 scope.go:122] "RemoveContainer" containerID="6910ddef44e0074579cfe27872a05bc54328fbdfcba6ae8535a51a0c984eddc5" Oct 29 23:43:05.764510 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.760207 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4" Oct 29 23:43:05.852846 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.851653 6507 scope.go:122] "RemoveContainer" containerID="6910ddef44e0074579cfe27872a05bc54328fbdfcba6ae8535a51a0c984eddc5" Oct 29 23:43:05.866511 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.863639 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4"] Oct 29 23:43:05.870717 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:43:05.870660 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6910ddef44e0074579cfe27872a05bc54328fbdfcba6ae8535a51a0c984eddc5\": container with ID starting with 6910ddef44e0074579cfe27872a05bc54328fbdfcba6ae8535a51a0c984eddc5 not found: ID does not exist" containerID="6910ddef44e0074579cfe27872a05bc54328fbdfcba6ae8535a51a0c984eddc5" Oct 29 23:43:05.871000 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.870967 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6910ddef44e0074579cfe27872a05bc54328fbdfcba6ae8535a51a0c984eddc5"} err="failed to get container status \"6910ddef44e0074579cfe27872a05bc54328fbdfcba6ae8535a51a0c984eddc5\": rpc error: code = NotFound desc = could not find container \"6910ddef44e0074579cfe27872a05bc54328fbdfcba6ae8535a51a0c984eddc5\": container with ID starting with 6910ddef44e0074579cfe27872a05bc54328fbdfcba6ae8535a51a0c984eddc5 not found: ID does not exist" Oct 29 23:43:05.871373 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:05.871350 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-809a6f59-kserve-6b46d5b77c-j4hj4"] Oct 29 23:43:06.766027 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:06.765983 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28596a2a-5fa3-47f3-bd47-1f68ee8e3fec" path="/var/lib/kubelet/pods/28596a2a-5fa3-47f3-bd47-1f68ee8e3fec/volumes" Oct 29 23:43:08.757841 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:08.757792 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:43:14.163396 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.163354 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4"] Oct 29 23:43:14.164218 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.164189 6507 cpu_manager.go:414] "containerMap: removing container" podUID="55a6367d-6371-4e1a-95b9-4a2e129c0399" containerName="main" Oct 29 23:43:14.164218 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.164217 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a6367d-6371-4e1a-95b9-4a2e129c0399" containerName="main" Oct 29 23:43:14.164535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.164255 6507 cpu_manager.go:414] "containerMap: removing container" podUID="18d09d9c-b1d3-4d4d-a0a2-1acc70371a79" containerName="revoke-keys" Oct 29 23:43:14.164535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.164264 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d09d9c-b1d3-4d4d-a0a2-1acc70371a79" containerName="revoke-keys" Oct 29 23:43:14.164535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.164287 6507 cpu_manager.go:414] "containerMap: removing container" podUID="28596a2a-5fa3-47f3-bd47-1f68ee8e3fec" containerName="main" Oct 29 23:43:14.164535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.164296 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28596a2a-5fa3-47f3-bd47-1f68ee8e3fec" containerName="main" Oct 29 23:43:14.164535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.164383 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="28596a2a-5fa3-47f3-bd47-1f68ee8e3fec" containerName="main" Oct 29 23:43:14.164535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.164397 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a6367d-6371-4e1a-95b9-4a2e129c0399" containerName="main" Oct 29 23:43:14.164535 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.164409 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d09d9c-b1d3-4d4d-a0a2-1acc70371a79" containerName="revoke-keys" Oct 29 23:43:14.167799 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.167778 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4" Oct 29 23:43:14.179843 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.179770 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:43:14.180000 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.179937 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-txsxx\"" Oct 29 23:43:14.190578 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.190536 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:43:14.190775 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.190591 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:43:14.192291 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.192258 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4"] Oct 29 23:43:14.201301 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.201259 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd94636-9409-4646-bb32-e6ceafb4b2c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd5795c1-4vkg4\" (UID: \"9cd94636-9409-4646-bb32-e6ceafb4b2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4" Oct 29 23:43:14.201510 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.201366 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd94636-9409-4646-bb32-e6ceafb4b2c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd5795c1-4vkg4\" (UID: \"9cd94636-9409-4646-bb32-e6ceafb4b2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4" Oct 29 23:43:14.201510 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.201417 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2xdt\" (UniqueName: \"kubernetes.io/projected/9cd94636-9409-4646-bb32-e6ceafb4b2c0-kube-api-access-s2xdt\") pod \"maas-api-revoke-sub-dd5795c1-4vkg4\" (UID: \"9cd94636-9409-4646-bb32-e6ceafb4b2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4" Oct 29 23:43:14.302016 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.301982 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd94636-9409-4646-bb32-e6ceafb4b2c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd5795c1-4vkg4\" (UID: \"9cd94636-9409-4646-bb32-e6ceafb4b2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4" Oct 29 23:43:14.302223 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.302065 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd94636-9409-4646-bb32-e6ceafb4b2c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd5795c1-4vkg4\" (UID: \"9cd94636-9409-4646-bb32-e6ceafb4b2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4" Oct 29 23:43:14.302223 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.302107 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2xdt\" (UniqueName: \"kubernetes.io/projected/9cd94636-9409-4646-bb32-e6ceafb4b2c0-kube-api-access-s2xdt\") pod \"maas-api-revoke-sub-dd5795c1-4vkg4\" (UID: \"9cd94636-9409-4646-bb32-e6ceafb4b2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4" Oct 29 23:43:14.302770 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.302745 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd94636-9409-4646-bb32-e6ceafb4b2c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd5795c1-4vkg4\" (UID: \"9cd94636-9409-4646-bb32-e6ceafb4b2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4" Oct 29 23:43:14.304314 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.304293 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd94636-9409-4646-bb32-e6ceafb4b2c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd5795c1-4vkg4\" (UID: \"9cd94636-9409-4646-bb32-e6ceafb4b2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4" Oct 29 23:43:14.323730 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.323699 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2xdt\" (UniqueName: \"kubernetes.io/projected/9cd94636-9409-4646-bb32-e6ceafb4b2c0-kube-api-access-s2xdt\") pod \"maas-api-revoke-sub-dd5795c1-4vkg4\" (UID: \"9cd94636-9409-4646-bb32-e6ceafb4b2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4" Oct 29 23:43:14.484857 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.484813 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4" Oct 29 23:43:14.779868 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.779777 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4"] Oct 29 23:43:14.783938 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:43:14.783899 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd94636_9409_4646_bb32_e6ceafb4b2c0.slice/crio-fe4cdfb429af73b36417cca91c126f42bde4c31792f8b0eac9268dbd700bb4d3 WatchSource:0}: Error finding container fe4cdfb429af73b36417cca91c126f42bde4c31792f8b0eac9268dbd700bb4d3: Status 404 returned error can't find the container with id fe4cdfb429af73b36417cca91c126f42bde4c31792f8b0eac9268dbd700bb4d3 Oct 29 23:43:14.861502 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:14.861449 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4" event={"ID":"9cd94636-9409-4646-bb32-e6ceafb4b2c0","Type":"ContainerStarted","Data":"fe4cdfb429af73b36417cca91c126f42bde4c31792f8b0eac9268dbd700bb4d3"} Oct 29 23:43:15.223371 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:43:15.223329 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:43:15.869497 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:15.869432 6507 generic.go:356] "Generic (PLEG): container finished" podID="9cd94636-9409-4646-bb32-e6ceafb4b2c0" containerID="466b4dafa045879df6272f47dd85a00a704fd2dd6601764616efb0c125bfdeaf" exitCode=0 Oct 29 23:43:15.869688 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:15.869517 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4" event={"ID":"9cd94636-9409-4646-bb32-e6ceafb4b2c0","Type":"ContainerDied","Data":"466b4dafa045879df6272f47dd85a00a704fd2dd6601764616efb0c125bfdeaf"} Oct 29 23:43:17.024199 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:17.024173 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4" Oct 29 23:43:17.129680 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:17.129596 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cd94636-9409-4646-bb32-e6ceafb4b2c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd94636-9409-4646-bb32-e6ceafb4b2c0-maas-api-xks-ca\") pod \"9cd94636-9409-4646-bb32-e6ceafb4b2c0\" (UID: \"9cd94636-9409-4646-bb32-e6ceafb4b2c0\") " Oct 29 23:43:17.129851 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:17.129694 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd94636-9409-4646-bb32-e6ceafb4b2c0-kube-api-access-s2xdt\" (UniqueName: \"kubernetes.io/projected/9cd94636-9409-4646-bb32-e6ceafb4b2c0-kube-api-access-s2xdt\") pod \"9cd94636-9409-4646-bb32-e6ceafb4b2c0\" (UID: \"9cd94636-9409-4646-bb32-e6ceafb4b2c0\") " Oct 29 23:43:17.129851 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:17.129719 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd94636-9409-4646-bb32-e6ceafb4b2c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd94636-9409-4646-bb32-e6ceafb4b2c0-maas-api-openshift-ca\") pod \"9cd94636-9409-4646-bb32-e6ceafb4b2c0\" (UID: \"9cd94636-9409-4646-bb32-e6ceafb4b2c0\") " Oct 29 23:43:17.130172 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:17.130145 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd94636-9409-4646-bb32-e6ceafb4b2c0-maas-api-openshift-ca" pod "9cd94636-9409-4646-bb32-e6ceafb4b2c0" (UID: "9cd94636-9409-4646-bb32-e6ceafb4b2c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:43:17.131933 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:17.131911 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd94636-9409-4646-bb32-e6ceafb4b2c0-maas-api-xks-ca" pod "9cd94636-9409-4646-bb32-e6ceafb4b2c0" (UID: "9cd94636-9409-4646-bb32-e6ceafb4b2c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:43:17.132003 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:17.131928 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd94636-9409-4646-bb32-e6ceafb4b2c0-kube-api-access-s2xdt" pod "9cd94636-9409-4646-bb32-e6ceafb4b2c0" (UID: "9cd94636-9409-4646-bb32-e6ceafb4b2c0"). InnerVolumeSpecName "kube-api-access-s2xdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:43:17.231181 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:17.231148 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2xdt\" (UniqueName: \"kubernetes.io/projected/9cd94636-9409-4646-bb32-e6ceafb4b2c0-kube-api-access-s2xdt\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:17.231181 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:17.231177 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd94636-9409-4646-bb32-e6ceafb4b2c0-maas-api-openshift-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:17.231181 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:17.231187 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd94636-9409-4646-bb32-e6ceafb4b2c0-maas-api-xks-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:17.881771 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:17.881740 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4" Oct 29 23:43:17.881948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:17.881771 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5795c1-4vkg4" event={"ID":"9cd94636-9409-4646-bb32-e6ceafb4b2c0","Type":"ContainerDied","Data":"fe4cdfb429af73b36417cca91c126f42bde4c31792f8b0eac9268dbd700bb4d3"} Oct 29 23:43:17.881948 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:17.881823 6507 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe4cdfb429af73b36417cca91c126f42bde4c31792f8b0eac9268dbd700bb4d3" Oct 29 23:43:25.191246 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:25.191204 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:43:26.358639 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.358600 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-ccp95"] Oct 29 23:43:26.359675 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.359644 6507 cpu_manager.go:414] "containerMap: removing container" podUID="9cd94636-9409-4646-bb32-e6ceafb4b2c0" containerName="revoke-keys" Oct 29 23:43:26.359675 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.359672 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd94636-9409-4646-bb32-e6ceafb4b2c0" containerName="revoke-keys" Oct 29 23:43:26.359824 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.359742 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd94636-9409-4646-bb32-e6ceafb4b2c0" containerName="revoke-keys" Oct 29 23:43:26.364101 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.364080 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:26.386951 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.386888 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 29 23:43:26.387235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.386969 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 23:43:26.387235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.386975 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 23:43:26.387235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.387034 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kxtx2\"" Oct 29 23:43:26.387235 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.387169 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 29 23:43:26.389755 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.389695 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-ccp95"] Oct 29 23:43:26.420106 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.420066 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lps8q\" (UniqueName: \"kubernetes.io/projected/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-kube-api-access-lps8q\") pod \"maas-api-5c48877c75-ccp95\" (UID: \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:26.420106 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.420109 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-openshift-service-ca\") pod \"maas-api-5c48877c75-ccp95\" (UID: \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:26.420333 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.420249 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-maas-api-tls\") pod \"maas-api-5c48877c75-ccp95\" (UID: \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:26.420333 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.420284 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-metrics-certs\") pod \"maas-api-5c48877c75-ccp95\" (UID: \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:26.521780 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.521740 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-maas-api-tls\") pod \"maas-api-5c48877c75-ccp95\" (UID: \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:26.521967 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.521784 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-metrics-certs\") pod \"maas-api-5c48877c75-ccp95\" (UID: \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:26.521967 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.521867 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lps8q\" (UniqueName: \"kubernetes.io/projected/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-kube-api-access-lps8q\") pod \"maas-api-5c48877c75-ccp95\" (UID: \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:26.521967 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.521901 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-openshift-service-ca\") pod \"maas-api-5c48877c75-ccp95\" (UID: \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:26.522652 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.522628 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-openshift-service-ca\") pod \"maas-api-5c48877c75-ccp95\" (UID: \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:26.524392 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.524357 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-maas-api-tls\") pod \"maas-api-5c48877c75-ccp95\" (UID: \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:26.524513 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.524391 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-metrics-certs\") pod \"maas-api-5c48877c75-ccp95\" (UID: \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:26.538011 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.537983 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lps8q\" (UniqueName: \"kubernetes.io/projected/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-kube-api-access-lps8q\") pod \"maas-api-5c48877c75-ccp95\" (UID: \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:26.678343 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.678251 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:26.978601 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.978230 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-ccp95"] Oct 29 23:43:26.984019 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:43:26.983988 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5cc8f9_9180_4f0d_be1f_5df44d64abec.slice/crio-c41832b3818cecfa58deade128e4de3d560b007ad0f233c3b824440358f106c6 WatchSource:0}: Error finding container c41832b3818cecfa58deade128e4de3d560b007ad0f233c3b824440358f106c6: Status 404 returned error can't find the container with id c41832b3818cecfa58deade128e4de3d560b007ad0f233c3b824440358f106c6 Oct 29 23:43:26.985875 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:26.985855 6507 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:43:27.025704 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:27.025663 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm"] Oct 29 23:43:27.030755 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:27.030720 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6pgm"] Oct 29 23:43:27.936445 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:27.936397 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" event={"ID":"1b5cc8f9-9180-4f0d-be1f-5df44d64abec","Type":"ContainerStarted","Data":"c41832b3818cecfa58deade128e4de3d560b007ad0f233c3b824440358f106c6"} Oct 29 23:43:28.751759 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:28.751682 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3150fb0-7b66-4555-80dc-f3f5f9619f7e" path="/var/lib/kubelet/pods/c3150fb0-7b66-4555-80dc-f3f5f9619f7e/volumes" Oct 29 23:43:28.943789 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:28.943756 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" event={"ID":"1b5cc8f9-9180-4f0d-be1f-5df44d64abec","Type":"ContainerStarted","Data":"53aa341f7222cdfcd2750a231aa730f438867799da9e45cd8be9bad3dbb3c61f"} Oct 29 23:43:28.944233 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:28.943930 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:28.978983 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:28.978921 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" podStartSLOduration=1.528619153 podStartE2EDuration="2.978906209s" totalImagesPullingTime="1.450287056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:43:28.977790049 +0000 UTC m=+1778.340794774" watchObservedRunningTime="2026-10-29 23:43:28.978906209 +0000 UTC m=+1778.341910931" Oct 29 23:43:33.576264 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.576216 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-ccp95"] Oct 29 23:43:33.576876 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.576489 6507 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" podUID="1b5cc8f9-9180-4f0d-be1f-5df44d64abec" containerName="maas-api" containerID="cri-o://53aa341f7222cdfcd2750a231aa730f438867799da9e45cd8be9bad3dbb3c61f" gracePeriod=30 Oct 29 23:43:33.582591 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.582569 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:33.618671 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.618634 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-c7dw5"] Oct 29 23:43:33.622096 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.622078 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" Oct 29 23:43:33.643910 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.643822 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-c7dw5"] Oct 29 23:43:33.696926 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.696887 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa22b99-08c8-4fb9-9bb8-79afbac9ce09-maas-api-tls\") pod \"maas-api-595cc648c-c7dw5\" (UID: \"1aa22b99-08c8-4fb9-9bb8-79afbac9ce09\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" Oct 29 23:43:33.697100 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.696995 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk4q2\" (UniqueName: \"kubernetes.io/projected/1aa22b99-08c8-4fb9-9bb8-79afbac9ce09-kube-api-access-lk4q2\") pod \"maas-api-595cc648c-c7dw5\" (UID: \"1aa22b99-08c8-4fb9-9bb8-79afbac9ce09\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" Oct 29 23:43:33.697100 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.697042 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa22b99-08c8-4fb9-9bb8-79afbac9ce09-openshift-service-ca\") pod \"maas-api-595cc648c-c7dw5\" (UID: \"1aa22b99-08c8-4fb9-9bb8-79afbac9ce09\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" Oct 29 23:43:33.697226 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.697197 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa22b99-08c8-4fb9-9bb8-79afbac9ce09-metrics-certs\") pod \"maas-api-595cc648c-c7dw5\" (UID: \"1aa22b99-08c8-4fb9-9bb8-79afbac9ce09\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" Oct 29 23:43:33.798605 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.798564 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa22b99-08c8-4fb9-9bb8-79afbac9ce09-maas-api-tls\") pod \"maas-api-595cc648c-c7dw5\" (UID: \"1aa22b99-08c8-4fb9-9bb8-79afbac9ce09\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" Oct 29 23:43:33.798804 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.798619 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk4q2\" (UniqueName: \"kubernetes.io/projected/1aa22b99-08c8-4fb9-9bb8-79afbac9ce09-kube-api-access-lk4q2\") pod \"maas-api-595cc648c-c7dw5\" (UID: \"1aa22b99-08c8-4fb9-9bb8-79afbac9ce09\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" Oct 29 23:43:33.798804 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.798650 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa22b99-08c8-4fb9-9bb8-79afbac9ce09-openshift-service-ca\") pod \"maas-api-595cc648c-c7dw5\" (UID: \"1aa22b99-08c8-4fb9-9bb8-79afbac9ce09\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" Oct 29 23:43:33.798804 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.798739 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa22b99-08c8-4fb9-9bb8-79afbac9ce09-metrics-certs\") pod \"maas-api-595cc648c-c7dw5\" (UID: \"1aa22b99-08c8-4fb9-9bb8-79afbac9ce09\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" Oct 29 23:43:33.799532 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.799503 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa22b99-08c8-4fb9-9bb8-79afbac9ce09-openshift-service-ca\") pod \"maas-api-595cc648c-c7dw5\" (UID: \"1aa22b99-08c8-4fb9-9bb8-79afbac9ce09\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" Oct 29 23:43:33.801187 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.801164 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa22b99-08c8-4fb9-9bb8-79afbac9ce09-metrics-certs\") pod \"maas-api-595cc648c-c7dw5\" (UID: \"1aa22b99-08c8-4fb9-9bb8-79afbac9ce09\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" Oct 29 23:43:33.801280 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.801197 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa22b99-08c8-4fb9-9bb8-79afbac9ce09-maas-api-tls\") pod \"maas-api-595cc648c-c7dw5\" (UID: \"1aa22b99-08c8-4fb9-9bb8-79afbac9ce09\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" Oct 29 23:43:33.815489 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.815437 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk4q2\" (UniqueName: \"kubernetes.io/projected/1aa22b99-08c8-4fb9-9bb8-79afbac9ce09-kube-api-access-lk4q2\") pod \"maas-api-595cc648c-c7dw5\" (UID: \"1aa22b99-08c8-4fb9-9bb8-79afbac9ce09\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" Oct 29 23:43:33.832639 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.832574 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:33.967338 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.967298 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" Oct 29 23:43:33.971305 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.971278 6507 generic.go:356] "Generic (PLEG): container finished" podID="1b5cc8f9-9180-4f0d-be1f-5df44d64abec" containerID="53aa341f7222cdfcd2750a231aa730f438867799da9e45cd8be9bad3dbb3c61f" exitCode=0 Oct 29 23:43:33.971428 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.971330 6507 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" Oct 29 23:43:33.971428 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.971365 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" event={"ID":"1b5cc8f9-9180-4f0d-be1f-5df44d64abec","Type":"ContainerDied","Data":"53aa341f7222cdfcd2750a231aa730f438867799da9e45cd8be9bad3dbb3c61f"} Oct 29 23:43:33.971428 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.971423 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-ccp95" event={"ID":"1b5cc8f9-9180-4f0d-be1f-5df44d64abec","Type":"ContainerDied","Data":"c41832b3818cecfa58deade128e4de3d560b007ad0f233c3b824440358f106c6"} Oct 29 23:43:33.971576 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.971452 6507 scope.go:122] "RemoveContainer" containerID="53aa341f7222cdfcd2750a231aa730f438867799da9e45cd8be9bad3dbb3c61f" Oct 29 23:43:33.981038 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.981006 6507 scope.go:122] "RemoveContainer" containerID="53aa341f7222cdfcd2750a231aa730f438867799da9e45cd8be9bad3dbb3c61f" Oct 29 23:43:33.982864 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:43:33.981887 6507 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53aa341f7222cdfcd2750a231aa730f438867799da9e45cd8be9bad3dbb3c61f\": container with ID starting with 53aa341f7222cdfcd2750a231aa730f438867799da9e45cd8be9bad3dbb3c61f not found: ID does not exist" containerID="53aa341f7222cdfcd2750a231aa730f438867799da9e45cd8be9bad3dbb3c61f" Oct 29 23:43:33.982864 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:33.981960 6507 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53aa341f7222cdfcd2750a231aa730f438867799da9e45cd8be9bad3dbb3c61f"} err="failed to get container status \"53aa341f7222cdfcd2750a231aa730f438867799da9e45cd8be9bad3dbb3c61f\": rpc error: code = NotFound desc = could not find container \"53aa341f7222cdfcd2750a231aa730f438867799da9e45cd8be9bad3dbb3c61f\": container with ID starting with 53aa341f7222cdfcd2750a231aa730f438867799da9e45cd8be9bad3dbb3c61f not found: ID does not exist" Oct 29 23:43:34.001023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.000989 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-openshift-service-ca\") pod \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\" (UID: \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\") " Oct 29 23:43:34.001234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.001037 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-kube-api-access-lps8q\" (UniqueName: \"kubernetes.io/projected/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-kube-api-access-lps8q\") pod \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\" (UID: \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\") " Oct 29 23:43:34.001234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.001121 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-metrics-certs\") pod \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\" (UID: \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\") " Oct 29 23:43:34.001234 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.001160 6507 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-maas-api-tls\") pod \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\" (UID: \"1b5cc8f9-9180-4f0d-be1f-5df44d64abec\") " Oct 29 23:43:34.001438 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.001409 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-openshift-service-ca" pod "1b5cc8f9-9180-4f0d-be1f-5df44d64abec" (UID: "1b5cc8f9-9180-4f0d-be1f-5df44d64abec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:43:34.001584 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.001559 6507 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-openshift-service-ca\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:34.003367 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.003342 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-maas-api-tls" pod "1b5cc8f9-9180-4f0d-be1f-5df44d64abec" (UID: "1b5cc8f9-9180-4f0d-be1f-5df44d64abec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:43:34.003507 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.003401 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-kube-api-access-lps8q" pod "1b5cc8f9-9180-4f0d-be1f-5df44d64abec" (UID: "1b5cc8f9-9180-4f0d-be1f-5df44d64abec"). InnerVolumeSpecName "kube-api-access-lps8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:43:34.003507 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.003447 6507 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-metrics-certs" pod "1b5cc8f9-9180-4f0d-be1f-5df44d64abec" (UID: "1b5cc8f9-9180-4f0d-be1f-5df44d64abec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:43:34.102166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.102056 6507 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lps8q\" (UniqueName: \"kubernetes.io/projected/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-kube-api-access-lps8q\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:34.102166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.102090 6507 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-metrics-certs\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:34.102166 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.102100 6507 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b5cc8f9-9180-4f0d-be1f-5df44d64abec-maas-api-tls\") on node \"ip-10-0-133-100.ec2.internal\" DevicePath \"\"" Oct 29 23:43:34.268016 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.267975 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-c7dw5"] Oct 29 23:43:34.272721 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:43:34.272687 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa22b99_08c8_4fb9_9bb8_79afbac9ce09.slice/crio-0ebaad00ea4ff261698f3781bab53d6dbe58f1999017e8ac8c6b72a294888592 WatchSource:0}: Error finding container 0ebaad00ea4ff261698f3781bab53d6dbe58f1999017e8ac8c6b72a294888592: Status 404 returned error can't find the container with id 0ebaad00ea4ff261698f3781bab53d6dbe58f1999017e8ac8c6b72a294888592 Oct 29 23:43:34.311204 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.311156 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-ccp95"] Oct 29 23:43:34.315686 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.315640 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-ccp95"] Oct 29 23:43:34.753262 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.753232 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b5cc8f9-9180-4f0d-be1f-5df44d64abec" path="/var/lib/kubelet/pods/1b5cc8f9-9180-4f0d-be1f-5df44d64abec/volumes" Oct 29 23:43:34.979798 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.979751 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" event={"ID":"1aa22b99-08c8-4fb9-9bb8-79afbac9ce09","Type":"ContainerStarted","Data":"306c012e5a4310a79b3f71294fce39165b7cb0c2b969e3c1cae9fe0732bf8c96"} Oct 29 23:43:34.980193 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.979825 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" event={"ID":"1aa22b99-08c8-4fb9-9bb8-79afbac9ce09","Type":"ContainerStarted","Data":"0ebaad00ea4ff261698f3781bab53d6dbe58f1999017e8ac8c6b72a294888592"} Oct 29 23:43:34.980193 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:34.979884 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" Oct 29 23:43:35.014275 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:35.014206 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" podStartSLOduration=1.592888668 podStartE2EDuration="2.014186026s" totalImagesPullingTime="421.297358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:43:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:43:35.012031123 +0000 UTC m=+1784.375035846" watchObservedRunningTime="2026-10-29 23:43:35.014186026 +0000 UTC m=+1784.377190752" Oct 29 23:43:35.166087 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:35.166042 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:43:40.995809 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:40.995777 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-c7dw5" Oct 29 23:43:43.264121 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:43.264082 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:43:45.327486 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:43:45.326283 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:43:53.432828 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:53.432793 6507 scope.go:122] "RemoveContainer" containerID="0ce50e9bfb5ced8e1f8b31c9f5e77e05a9fc793ae1699ab77e90a7d7543881de" Oct 29 23:43:58.581819 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:43:58.581780 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:44:15.226311 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:44:15.226266 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:44:26.277791 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:26.277751 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5dnd2"] Oct 29 23:44:37.371422 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:37.371368 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9b4f55d59-srl67_ceac094c-fb24-4f92-906c-46d6df96dcda/manager/0.log" Oct 29 23:44:38.339035 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:38.338994 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-765946bc59-h74cr_b9c239c8-1cc2-408e-bfda-95be8f2da85f/manager/0.log" Oct 29 23:44:39.313090 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:39.313062 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bb7fb9d97-7bs76_aee652fc-2f03-42d6-8ad9-ef6169e20f95/copy-manifests/0.log" Oct 29 23:44:39.344811 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:39.344776 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bb7fb9d97-7bs76_aee652fc-2f03-42d6-8ad9-ef6169e20f95/manager/0.log" Oct 29 23:44:41.046806 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:41.046766 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-c7dw5_1aa22b99-08c8-4fb9-9bb8-79afbac9ce09/maas-api/0.log" Oct 29 23:44:41.431998 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:41.431898 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-qhdj8_18d09d9c-b1d3-4d4d-a0a2-1acc70371a79/revoke-keys/0.log" Oct 29 23:44:41.625845 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:41.625816 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-tw6r6_56e78642-43a0-4069-9c41-9ce4126fd145/revoke-keys/0.log" Oct 29 23:44:41.995520 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:41.995490 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cc36b4a1-jdsdh_efab58b1-884e-4508-9ee9-3ca2d57d8680/revoke-keys/0.log" Oct 29 23:44:42.188583 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:42.188554 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-5w5rz_9d1de21b-4915-45ea-aa3b-552f45d473ef/revoke-keys/0.log" Oct 29 23:44:42.383988 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:42.383919 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-bvjwt_11b18d64-a832-43f3-9dd7-a33f06a96422/revoke-keys/0.log" Oct 29 23:44:42.578894 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:42.578865 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dd5795c1-4vkg4_9cd94636-9409-4646-bb32-e6ceafb4b2c0/revoke-keys/0.log" Oct 29 23:44:44.764433 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:44.764388 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-q69nr_8d39ed61-cd5e-4610-ba11-365ee048c1b5/manager/0.log" Oct 29 23:44:44.978026 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:44.977992 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-xtrvr_69a9fdeb-90d5-4e2f-8655-c7cd6f5c462d/manager/0.log" Oct 29 23:44:45.222966 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:44:45.222930 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:44:45.667057 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:45.666981 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-5dnd2_1aa1f873-3de0-4311-abe5-27cf61c905fe/limitador/0.log" Oct 29 23:44:45.867393 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:45.867362 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-l89qp_cec1507f-b47f-4885-a6a5-3fcaf3a4adff/manager/0.log" Oct 29 23:44:46.686872 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:46.686845 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-557c4f746fxgh9l_d2ca8769-66ad-496c-aed6-e91aed841fa7/istio-proxy/0.log" Oct 29 23:44:46.884780 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:46.884746 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zhd8v_5fb4ff80-7ad5-4be5-9f2f-868fea0de363/discovery/0.log" Oct 29 23:44:47.087799 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:47.087730 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cbcb88b9-dbxcv_3f783198-516a-4ada-90be-3f53da4e8cb1/kube-auth-proxy/0.log" Oct 29 23:44:47.675077 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:47.675051 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-8gxl6_b9e3a005-5ced-4440-814c-f2713230bd95/istio-proxy/0.log" Oct 29 23:44:47.884903 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:47.884873 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-9phht_f116d226-bd4f-41bd-aba4-06958eccb990/istio-proxy/0.log" Oct 29 23:44:48.086984 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:48.086949 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-bsqhf_7084c3a6-6454-456d-a6ad-06f58bf3e4e0/istio-proxy/1.log" Oct 29 23:44:49.113748 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:49.113719 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-zf2k4_cc74a2a4-ee89-46f7-84e9-6e6744b31fb6/istio-proxy/0.log" Oct 29 23:44:50.332822 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:50.332789 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-bsqhf_7084c3a6-6454-456d-a6ad-06f58bf3e4e0/istio-proxy/0.log" Oct 29 23:44:51.081498 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:51.081448 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-z7xww_d8489303-190c-4dcf-8deb-125ecb21210c/main/0.log" Oct 29 23:44:51.673006 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:51.672972 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x_64bb8724-5ff7-4382-bec8-a81679b219fb/main/0.log" Oct 29 23:44:52.064064 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:52.064035 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv_1020c327-0d07-4228-8124-ce2b9d0413ba/main/0.log" Oct 29 23:44:53.463083 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:53.463052 6507 scope.go:122] "RemoveContainer" containerID="9312838268e0c81f4fff3c18a9978134384dff9e9540b5400f04bfba5ca8d97a" Oct 29 23:44:53.471689 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:44:53.471658 6507 scope.go:122] "RemoveContainer" containerID="6e7511ada42ece01b5bf49293cd17a60f24355a0289f7abd87f25a5ff939d550" Oct 29 23:45:02.259559 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:02.259521 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-9b4f55d59-srl67_ceac094c-fb24-4f92-906c-46d6df96dcda/manager/0.log" Oct 29 23:45:03.128368 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:03.128335 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-765946bc59-h74cr_b9c239c8-1cc2-408e-bfda-95be8f2da85f/manager/0.log" Oct 29 23:45:04.028200 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:04.028171 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bb7fb9d97-7bs76_aee652fc-2f03-42d6-8ad9-ef6169e20f95/copy-manifests/0.log" Oct 29 23:45:04.062210 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:04.062178 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bb7fb9d97-7bs76_aee652fc-2f03-42d6-8ad9-ef6169e20f95/manager/0.log" Oct 29 23:45:05.473856 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:05.473816 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-c7dw5_1aa22b99-08c8-4fb9-9bb8-79afbac9ce09/maas-api/0.log" Oct 29 23:45:06.056088 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:06.056055 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-qhdj8_18d09d9c-b1d3-4d4d-a0a2-1acc70371a79/revoke-keys/0.log" Oct 29 23:45:06.239846 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:06.239814 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-tw6r6_56e78642-43a0-4069-9c41-9ce4126fd145/revoke-keys/0.log" Oct 29 23:45:06.616060 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:06.616030 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cc36b4a1-jdsdh_efab58b1-884e-4508-9ee9-3ca2d57d8680/revoke-keys/0.log" Oct 29 23:45:06.795820 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:06.795793 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-5w5rz_9d1de21b-4915-45ea-aa3b-552f45d473ef/revoke-keys/0.log" Oct 29 23:45:06.986926 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:06.986899 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-bvjwt_11b18d64-a832-43f3-9dd7-a33f06a96422/revoke-keys/0.log" Oct 29 23:45:07.171832 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:07.171803 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dd5795c1-4vkg4_9cd94636-9409-4646-bb32-e6ceafb4b2c0/revoke-keys/0.log" Oct 29 23:45:08.936107 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:08.936079 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-q69nr_8d39ed61-cd5e-4610-ba11-365ee048c1b5/manager/0.log" Oct 29 23:45:09.154945 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:09.154915 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-xtrvr_69a9fdeb-90d5-4e2f-8655-c7cd6f5c462d/manager/0.log" Oct 29 23:45:09.797848 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:09.797812 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-5dnd2_1aa1f873-3de0-4311-abe5-27cf61c905fe/limitador/0.log" Oct 29 23:45:09.994861 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:09.994829 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-l89qp_cec1507f-b47f-4885-a6a5-3fcaf3a4adff/manager/0.log" Oct 29 23:45:10.608127 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:10.608096 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-557c4f746fxgh9l_d2ca8769-66ad-496c-aed6-e91aed841fa7/istio-proxy/0.log" Oct 29 23:45:10.801152 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:10.801122 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zhd8v_5fb4ff80-7ad5-4be5-9f2f-868fea0de363/discovery/0.log" Oct 29 23:45:11.001773 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:11.001736 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cbcb88b9-dbxcv_3f783198-516a-4ada-90be-3f53da4e8cb1/kube-auth-proxy/0.log" Oct 29 23:45:11.573030 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:11.573000 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-8gxl6_b9e3a005-5ced-4440-814c-f2713230bd95/istio-proxy/0.log" Oct 29 23:45:11.775066 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:11.775037 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-9phht_f116d226-bd4f-41bd-aba4-06958eccb990/istio-proxy/0.log" Oct 29 23:45:11.959777 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:11.959747 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-bsqhf_7084c3a6-6454-456d-a6ad-06f58bf3e4e0/istio-proxy/1.log" Oct 29 23:45:12.027018 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:12.026987 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt"] Oct 29 23:45:12.035734 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:12.035705 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bvjwt"] Oct 29 23:45:12.752428 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:12.752397 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b18d64-a832-43f3-9dd7-a33f06a96422" path="/var/lib/kubelet/pods/11b18d64-a832-43f3-9dd7-a33f06a96422/volumes" Oct 29 23:45:12.993386 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:12.993356 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-zf2k4_cc74a2a4-ee89-46f7-84e9-6e6744b31fb6/istio-proxy/0.log" Oct 29 23:45:14.555086 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:14.555050 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-z7xww_d8489303-190c-4dcf-8deb-125ecb21210c/main/0.log" Oct 29 23:45:15.170840 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:15.170802 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8slp7x_64bb8724-5ff7-4382-bec8-a81679b219fb/main/0.log" Oct 29 23:45:15.224283 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:45:15.224243 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:45:15.572572 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:15.572541 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-hlmsv_1020c327-0d07-4228-8124-ce2b9d0413ba/main/0.log" Oct 29 23:45:28.024351 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:28.024315 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz"] Oct 29 23:45:28.033690 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:28.033653 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5w5rz"] Oct 29 23:45:28.751237 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:28.751200 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d1de21b-4915-45ea-aa3b-552f45d473ef" path="/var/lib/kubelet/pods/9d1de21b-4915-45ea-aa3b-552f45d473ef/volumes" Oct 29 23:45:45.224323 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:45:45.224235 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:45:53.201227 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:53.201192 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-765946bc59-h74cr_b9c239c8-1cc2-408e-bfda-95be8f2da85f/manager/0.log" Oct 29 23:45:53.580692 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:45:53.580610 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-765946bc59-h74cr_b9c239c8-1cc2-408e-bfda-95be8f2da85f/manager/0.log" Oct 29 23:46:15.222372 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:46:15.222334 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:46:21.023470 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:21.023425 6507 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh"] Oct 29 23:46:21.032793 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:21.032762 6507 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc36b4a1-jdsdh"] Oct 29 23:46:22.751888 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:22.751855 6507 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efab58b1-884e-4508-9ee9-3ca2d57d8680" path="/var/lib/kubelet/pods/efab58b1-884e-4508-9ee9-3ca2d57d8680/volumes" Oct 29 23:46:40.961517 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:40.961477 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-765946bc59-h74cr_b9c239c8-1cc2-408e-bfda-95be8f2da85f/manager/0.log" Oct 29 23:46:41.305666 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:41.305588 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-765946bc59-h74cr_b9c239c8-1cc2-408e-bfda-95be8f2da85f/manager/0.log" Oct 29 23:46:45.223725 ip-10-0-133-100 kubenswrapper[6507]: E1029 23:46:45.223678 6507 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mmg8f" containerName="extractor" Oct 29 23:46:47.287721 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:47.287688 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xcszc_254e83a6-ab81-4b6b-b5ce-0538ccaede49/global-pull-secret-syncer/0.log" Oct 29 23:46:47.906406 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:47.906374 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-klctw_e6cba05f-3979-445f-a517-da054562cc20/konnectivity-agent/0.log" Oct 29 23:46:48.164390 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:48.164307 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-100.ec2.internal_3a7e411141454709a32dcea46ca88fb7/haproxy/0.log" Oct 29 23:46:53.500620 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:53.500588 6507 scope.go:122] "RemoveContainer" containerID="0c0b4875d44e351e73f751685ec921068b29fa4935c65222470ff7dc83663210" Oct 29 23:46:53.509145 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:53.509124 6507 scope.go:122] "RemoveContainer" containerID="c853d88558d758e2a351989567405cd7534af14d0e01f9835f054342e0edca60" Oct 29 23:46:55.359800 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:55.359767 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-q69nr_8d39ed61-cd5e-4610-ba11-365ee048c1b5/manager/0.log" Oct 29 23:46:55.483818 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:55.483788 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-xtrvr_69a9fdeb-90d5-4e2f-8655-c7cd6f5c462d/manager/0.log" Oct 29 23:46:55.908593 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:55.908564 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-5dnd2_1aa1f873-3de0-4311-abe5-27cf61c905fe/limitador/0.log" Oct 29 23:46:56.053042 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:56.053015 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-l89qp_cec1507f-b47f-4885-a6a5-3fcaf3a4adff/manager/0.log" Oct 29 23:46:56.942802 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:56.942768 6507 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m"] Oct 29 23:46:56.943557 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:56.943407 6507 cpu_manager.go:414] "containerMap: removing container" podUID="1b5cc8f9-9180-4f0d-be1f-5df44d64abec" containerName="maas-api" Oct 29 23:46:56.943557 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:56.943426 6507 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b5cc8f9-9180-4f0d-be1f-5df44d64abec" containerName="maas-api" Oct 29 23:46:56.943557 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:56.943530 6507 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b5cc8f9-9180-4f0d-be1f-5df44d64abec" containerName="maas-api" Oct 29 23:46:56.948165 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:56.948146 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:56.959915 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:56.959892 6507 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gbnpg\"/\"default-dockercfg-kv5q7\"" Oct 29 23:46:56.971521 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:56.971486 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m"] Oct 29 23:46:57.041907 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.041875 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gbnpg\"/\"openshift-service-ca.crt\"" Oct 29 23:46:57.042172 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.042154 6507 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gbnpg\"/\"kube-root-ca.crt\"" Oct 29 23:46:57.042255 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.042171 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/72508bcd-3a46-4047-8d54-53e631cf49b3-proc\") pod \"perf-node-gather-daemonset-zgx7m\" (UID: \"72508bcd-3a46-4047-8d54-53e631cf49b3\") " pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:57.042255 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.042231 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tb75\" (UniqueName: \"kubernetes.io/projected/72508bcd-3a46-4047-8d54-53e631cf49b3-kube-api-access-7tb75\") pod \"perf-node-gather-daemonset-zgx7m\" (UID: \"72508bcd-3a46-4047-8d54-53e631cf49b3\") " pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:57.042338 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.042302 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72508bcd-3a46-4047-8d54-53e631cf49b3-sys\") pod \"perf-node-gather-daemonset-zgx7m\" (UID: \"72508bcd-3a46-4047-8d54-53e631cf49b3\") " pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:57.042387 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.042375 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72508bcd-3a46-4047-8d54-53e631cf49b3-lib-modules\") pod \"perf-node-gather-daemonset-zgx7m\" (UID: \"72508bcd-3a46-4047-8d54-53e631cf49b3\") " pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:57.042423 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.042414 6507 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/72508bcd-3a46-4047-8d54-53e631cf49b3-podres\") pod \"perf-node-gather-daemonset-zgx7m\" (UID: \"72508bcd-3a46-4047-8d54-53e631cf49b3\") " pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:57.143650 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.143615 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/72508bcd-3a46-4047-8d54-53e631cf49b3-proc\") pod \"perf-node-gather-daemonset-zgx7m\" (UID: \"72508bcd-3a46-4047-8d54-53e631cf49b3\") " pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:57.143839 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.143663 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tb75\" (UniqueName: \"kubernetes.io/projected/72508bcd-3a46-4047-8d54-53e631cf49b3-kube-api-access-7tb75\") pod \"perf-node-gather-daemonset-zgx7m\" (UID: \"72508bcd-3a46-4047-8d54-53e631cf49b3\") " pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:57.143839 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.143680 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72508bcd-3a46-4047-8d54-53e631cf49b3-sys\") pod \"perf-node-gather-daemonset-zgx7m\" (UID: \"72508bcd-3a46-4047-8d54-53e631cf49b3\") " pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:57.143839 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.143726 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72508bcd-3a46-4047-8d54-53e631cf49b3-lib-modules\") pod \"perf-node-gather-daemonset-zgx7m\" (UID: \"72508bcd-3a46-4047-8d54-53e631cf49b3\") " pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:57.143839 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.143750 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/72508bcd-3a46-4047-8d54-53e631cf49b3-proc\") pod \"perf-node-gather-daemonset-zgx7m\" (UID: \"72508bcd-3a46-4047-8d54-53e631cf49b3\") " pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:57.143839 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.143762 6507 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/72508bcd-3a46-4047-8d54-53e631cf49b3-podres\") pod \"perf-node-gather-daemonset-zgx7m\" (UID: \"72508bcd-3a46-4047-8d54-53e631cf49b3\") " pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:57.143839 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.143824 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72508bcd-3a46-4047-8d54-53e631cf49b3-sys\") pod \"perf-node-gather-daemonset-zgx7m\" (UID: \"72508bcd-3a46-4047-8d54-53e631cf49b3\") " pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:57.144089 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.143845 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/72508bcd-3a46-4047-8d54-53e631cf49b3-podres\") pod \"perf-node-gather-daemonset-zgx7m\" (UID: \"72508bcd-3a46-4047-8d54-53e631cf49b3\") " pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:57.144089 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.143827 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72508bcd-3a46-4047-8d54-53e631cf49b3-lib-modules\") pod \"perf-node-gather-daemonset-zgx7m\" (UID: \"72508bcd-3a46-4047-8d54-53e631cf49b3\") " pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:57.162313 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.162285 6507 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tb75\" (UniqueName: \"kubernetes.io/projected/72508bcd-3a46-4047-8d54-53e631cf49b3-kube-api-access-7tb75\") pod \"perf-node-gather-daemonset-zgx7m\" (UID: \"72508bcd-3a46-4047-8d54-53e631cf49b3\") " pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:57.261108 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.261027 6507 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:57.551945 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:57.551869 6507 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m"] Oct 29 23:46:57.555567 ip-10-0-133-100 kubenswrapper[6507]: W1029 23:46:57.555534 6507 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod72508bcd_3a46_4047_8d54_53e631cf49b3.slice/crio-1c886577c88530300e247714e30d4aeb1a62f2d7b82f45737174fe0c07e8012c WatchSource:0}: Error finding container 1c886577c88530300e247714e30d4aeb1a62f2d7b82f45737174fe0c07e8012c: Status 404 returned error can't find the container with id 1c886577c88530300e247714e30d4aeb1a62f2d7b82f45737174fe0c07e8012c Oct 29 23:46:58.106901 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:58.106862 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" event={"ID":"72508bcd-3a46-4047-8d54-53e631cf49b3","Type":"ContainerStarted","Data":"52d404bae0eebaec621e67cf6fd1815eaee850a1a748106b73d7c2934213e087"} Oct 29 23:46:58.107315 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:58.106918 6507 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" event={"ID":"72508bcd-3a46-4047-8d54-53e631cf49b3","Type":"ContainerStarted","Data":"1c886577c88530300e247714e30d4aeb1a62f2d7b82f45737174fe0c07e8012c"} Oct 29 23:46:58.107315 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:58.106938 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:46:58.141816 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:58.141760 6507 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" podStartSLOduration=2.141744582 podStartE2EDuration="2.141744582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:46:58.14048543 +0000 UTC m=+1987.503490147" watchObservedRunningTime="2026-10-29 23:46:58.141744582 +0000 UTC m=+1987.504749296" Oct 29 23:46:58.465359 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:58.465322 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qs5th_73623961-3ffb-4492-a2f4-74a5c4b9551e/kube-state-metrics/0.log" Oct 29 23:46:58.590700 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:58.590664 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qs5th_73623961-3ffb-4492-a2f4-74a5c4b9551e/kube-rbac-proxy-main/0.log" Oct 29 23:46:58.707394 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:58.707355 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qs5th_73623961-3ffb-4492-a2f4-74a5c4b9551e/kube-rbac-proxy-self/0.log" Oct 29 23:46:59.054559 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:59.054530 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8qc8d_8cccc782-d044-4032-ba8b-8f4d4cdab41b/node-exporter/0.log" Oct 29 23:46:59.171379 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:59.171347 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8qc8d_8cccc782-d044-4032-ba8b-8f4d4cdab41b/kube-rbac-proxy/0.log" Oct 29 23:46:59.287141 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:46:59.287111 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8qc8d_8cccc782-d044-4032-ba8b-8f4d4cdab41b/init-textfile/0.log" Oct 29 23:47:00.067617 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:00.067580 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-j467d_7e3f3838-8917-45a1-a496-652392caa72c/kube-rbac-proxy-main/0.log" Oct 29 23:47:00.185820 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:00.185789 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-j467d_7e3f3838-8917-45a1-a496-652392caa72c/kube-rbac-proxy-self/0.log" Oct 29 23:47:00.304082 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:00.304050 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-j467d_7e3f3838-8917-45a1-a496-652392caa72c/openshift-state-metrics/0.log" Oct 29 23:47:00.436805 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:00.436773 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2c95f4b0-5da7-4d43-ab9e-c55da3688e89/prometheus/0.log" Oct 29 23:47:00.555922 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:00.555853 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2c95f4b0-5da7-4d43-ab9e-c55da3688e89/config-reloader/0.log" Oct 29 23:47:00.674793 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:00.674764 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2c95f4b0-5da7-4d43-ab9e-c55da3688e89/thanos-sidecar/0.log" Oct 29 23:47:00.793114 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:00.793035 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2c95f4b0-5da7-4d43-ab9e-c55da3688e89/kube-rbac-proxy-web/0.log" Oct 29 23:47:00.912086 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:00.912057 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2c95f4b0-5da7-4d43-ab9e-c55da3688e89/kube-rbac-proxy/0.log" Oct 29 23:47:01.042287 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:01.042256 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2c95f4b0-5da7-4d43-ab9e-c55da3688e89/kube-rbac-proxy-thanos/0.log" Oct 29 23:47:01.175018 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:01.174929 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2c95f4b0-5da7-4d43-ab9e-c55da3688e89/init-config-reloader/0.log" Oct 29 23:47:04.122959 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:04.122931 6507 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gbnpg/perf-node-gather-daemonset-zgx7m" Oct 29 23:47:09.445889 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:09.445856 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5bzwq_916cb471-df04-4560-aaea-7e9d27f41897/dns/0.log" Oct 29 23:47:09.561876 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:09.561851 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5bzwq_916cb471-df04-4560-aaea-7e9d27f41897/kube-rbac-proxy/0.log" Oct 29 23:47:09.836093 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:09.835999 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-557c4f746fxgh9l_d2ca8769-66ad-496c-aed6-e91aed841fa7/istio-proxy/0.log" Oct 29 23:47:10.244886 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:10.244853 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f456x_76e09503-45e8-4e94-9a3b-a5d291a3990f/dns-node-resolver/0.log" Oct 29 23:47:10.834857 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:10.834823 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zhd8v_5fb4ff80-7ad5-4be5-9f2f-868fea0de363/discovery/0.log" Oct 29 23:47:10.868877 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:10.868842 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cbcb88b9-dbxcv_3f783198-516a-4ada-90be-3f53da4e8cb1/kube-auth-proxy/0.log" Oct 29 23:47:11.060156 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:11.060120 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-8gxl6_b9e3a005-5ced-4440-814c-f2713230bd95/istio-proxy/0.log" Oct 29 23:47:11.123942 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:11.123867 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-9phht_f116d226-bd4f-41bd-aba4-06958eccb990/istio-proxy/0.log" Oct 29 23:47:11.185960 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:11.185919 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-bsqhf_7084c3a6-6454-456d-a6ad-06f58bf3e4e0/istio-proxy/0.log" Oct 29 23:47:11.187023 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:11.187004 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-bsqhf_7084c3a6-6454-456d-a6ad-06f58bf3e4e0/istio-proxy/1.log" Oct 29 23:47:11.471374 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:11.471349 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-zf2k4_cc74a2a4-ee89-46f7-84e9-6e6744b31fb6/istio-proxy/0.log" Oct 29 23:47:11.691823 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:11.691794 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tr7bl_6170bf16-e81d-40fd-a4a0-449213b94d19/node-ca/0.log" Oct 29 23:47:13.641195 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:13.641162 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-557c4f746fxgh9l_d2ca8769-66ad-496c-aed6-e91aed841fa7/istio-proxy/0.log" Oct 29 23:47:13.944316 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:13.944285 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-557c4f746fxgh9l_d2ca8769-66ad-496c-aed6-e91aed841fa7/istio-proxy/0.log" Oct 29 23:47:15.118729 ip-10-0-133-100 kubenswrapper[6507]: I1029 23:47:15.118690 6507 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zhd8v_5fb4ff80-7ad5-4be5-9f2f-868fea0de363/discovery/0.log"